- ceph-users - lists.ceph.io

Re: cephadm: Replacing OSD in mixed setup (block and block.db) is quite messy
by Anthony D'Atri 11 Mar '26

11 Mar '26

> > - 36 data disks, registered as mpath-devices > - 2 NVMEs, which act as block.db for all 36 spinning disks. That’s an unusually high ratio. Which model exactly are the NVMe SSDs? And why are you using multipathing with the HDDs? > I hacked a workaround, which removes the last conditional check (occupied_slots < fast_slots_per_device) and hard-coded the expected size of each block.db in my deployment file: Honestly that’s what I often do, especially when applying the orch service to systems with OSDs that were previously deployed under a different scheme. Hybrid OSDs can be messy. > service_type: osd > service_id: delta2024_osd > service_name: osd.delta2024_osd > placement: > label: delta2024 > spec: > block_db_size: 397G > data_devices: > rotational: 1 > size: '15T:' > db_devices: > rotational: 0 > encrypted: true > filter_logic: AND > objectstore: bluestore > > This gives the expected results. > In my opinion, cephadm sends a wrong „ceph-volume lvm batch“ command to the osd-node. It should always include all of the disks, since running it is promised to be idempotent. With the full list of disks, ceph-volume should be able to calculate correct slots for block.db. > > Did I find a bug here or is this expected behavior? Good question. Bitte enter a ticket at tracker.ceph.com <http://tracker.ceph.com/>. > > > > --------------------------- > M.Sc Alex Walender > Institut für Bio- und Geowissenschaften > IBG 5 - Computergestützte Metagenomik / > de.NBI Cloud Site Bielefeld > Büro : Universität Bielefeld (UHG), N7-101 > Tel. : +49-521-106-2907 > > Forschungszentrum Jülich GmbH > 52425 Jülich > Sitz der Gesellschaft: Jülich > Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Stefan Müller > Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), > Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: tentacle 20.2.1 QE validation status
by Ilya Dryomov 11 Mar '26

11 Mar '26

On Mon, Mar 9, 2026 at 11:38 PM Ilya Dryomov via ceph-users <ceph-users(a)ceph.io> wrote: > > On Mon, Mar 9, 2026 at 7:50 PM Yuri Weinstein via Dev <dev(a)ceph.io> wrote: > > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/74838#note-4 > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Radek, Laura, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > Hi Yuri, > > Could you please do another rerun for rbd? rbd approved based on https://pulpito.ceph.com/yuriw-2026-03-10_15:51:59-rbd-tentacle-release-dis… Thanks, Ilya

1 0

Re: Can rbd sparsify be run on a running VM's disk?
by Alwin Antreich 11 Mar '26

11 Mar '26

Hi Steve, I'd advise against using sparsify, especially on any mapped RBD (running VM). We had a support case once where this operation on a live VM caused an OSD lockup, halting the operation for other clients. Sparsify really tries to find 0-filled objects that can be removed. It's best to stick with fstrim (or equivalent) in the guest. rbd du estimates size cheaply by counting the number of objects and multiplying that count by the set object size. Therefore it diverges from the size you see of the guest's filesystem. Some additional points may contribute to the difference: * An OSD has a 4K (SSD) / 64K (HDD) threshold (bluestore_min_alloc_size) for discard operations, meaning any discard smaller than this threshold is ignored. * Disk encryption often doesn't clear block for security. * EXT4 uses an in-memory bitmap that does not trim blocks if they haven't been overwritten before. This safeguards against wear on consumer SSDs but has the disadvantage that a storage move will thick-allocate the disk and an fstrim isn't clearing it. The only option is to write a big file and delete it or reboot. See: https://forum.proxmox.com/threads/help-with-trim-on-virtio-scsi-single.1238… I hope this helps. Cheers, Alwin On Tue, 10 Mar 2026 at 14:59, Steve Yates via ceph-users <ceph-users(a)ceph.io> wrote: > Question: Can rbd sparsify safely be run on a running VM's disk, in Ceph > 19/Squid? Or must the VM be shut down first? I tried looking online but > am not finding a direct answer. > > More info: I'm looking into why "rbd du" shows usage inconsistent with > "ceph df" or Proxmox's "usage" displays (Proxmox thread< > https://forum.proxmox.com/threads/ceph-rbd-du-shows-usage-2-4x-higher-than-…>). > Sparsify on a stopped VM reduces "rbd du" usage, and the number of objects, > but doesn't change the amount of space used according to Ceph and PVE. I > am wondering if sparsify can be safely scripted. Or if I should just > ignore "rbd du" USED amounts. > > Thanks. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Alwin Antreich Head of Training and Proxmox Services Want to meet: https://calendar.app.google/MuA2isCGnh8xBb657 croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> TOP 100 Innovator Award <https://croit.io/blog/croit-receives-top-100-seal> Winner by compamedia Technology Fast50 Award <https://croit.io/blog/deloitte-technology-fast-50-award> Winner by Deloitte

1 0

Ceph Reef (18.2.4) OSD fails to start with “Permission denied” after Ubuntu 20.04 -> 22.04 upgrade (cephadm)
by Jay See 11 Mar '26

11 Mar '26

Hi all, After upgrading a Ceph Reef (18.2.4) node from Ubuntu 20.04 (kernel 5.4.0) to Ubuntu 22.04 (kernel 5.15.0-94), a cephadm-managed OSD fails to start with a “Permission denied” error. I’ve checked for related bugs, including *tracker.ceph.com/issues/66989#note-8 <http://tracker.ceph.com/issues/66989#note-8>* , and even tried adding "*--security-opt seccomp=unconfined*", but the issue persists. There are more than one OSD, just sharing the details of one of the OSD here. Syslogs from a failed OSD start are available here: *https://paste.ubuntu.com/p/8n52TJdcSt/ <https://paste.ubuntu.com/p/8n52TJdcSt/>. * ~$ ceph orch ps --hostname h006 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID osd.9 h006 error 22h ago 4y - 4096M <unknown> <unknown> Can anyone point me to a solution, or to previous discussions or bugs addressing this problem? Thanks in advance, Jay.

1 0

cephadm: Replacing OSD in mixed setup (block and block.db) is quite messy
by Alex Walender 11 Mar '26

11 Mar '26

Hey friends, Our production system is still running ceph with manual Debian packages. Since I’d like to migrate to a deployment based on cephadm, I experimented the migration and usage of day 1 ops on our development environment. Both systems run with v18.2.7. The migration of the dev-system went well. The only headache I have is with replacing OSDs (simulating the replacement of a faulty disk). The osd-node has the following setup: - 36 data disks, registered as mpath-devices - 2 NVMEs, which act as block.db for all 36 spinning disks. Therefore each NVME holds 18 LVs. For the deployment I used a very simple file: ``` service_type: osd service_id: delta2024_osd placement: label: delta2024 spec: data_devices: size: '15T:' rotational: 1 db_devices: size: '6T:' rotational: 0 objectstore: bluestore encrypted: true ``` This deploys all osds as expected. On cephadm-logs, I see the „ceph-volume lvm batch“ command, including all 36 disks as arguments. Now I want to simulate the replacement of a failing disk: - Set the deployment to unmanaged via „ceph orch set-unmanaged …“ - Zap one disk with „ceph orch osd rm 56 —zap --replace“ - „vgs“ show, that one of the NVMEs has space again. - „ceph orch device ls —wide —refresh“ shows the zapped disk as available again - Set the deployment to managed again „ceph orch set-managed …“ - I expect, that the OSD gets re-deployed again, but nothing happens. This time, cephadm sends a shorter command to my osd-node: „ceph-volume lvm batch --auto /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 --no-systemd —yes“ This exits with the message, that none of the passed fast-devices are available. After digging in the code for ceph-volume, I may found an issue: Since only one disk is included in the batch call (/dev/mapper/mpathz), ceph-volume tries to calculate a whole NVME disk for its block.db. This, of course, doesn’t work. It has no clue on how to properly slice a LV for the new block.db on an existing NVME drive given the passed arguments. The check that fails is this snippet (ceph_volume/devices/lvm/batch.py): while abs_size <= free_size and len(ret) < new_osds and occupied_slots < fast_slots_per_device: free_size -= abs_size.b occupied_slots += 1 ret.append((dev.path, relative_size, abs_size, requested_slots)) When using an empty node, occupied_slots = 0, and fast_slots_per_device = 18. Therefore the deployment works. When only replacing one OSD, occupied_slots = 17 (because one LV got zapped) and fast_slots_per_device = 1. Therefore the last conditional check is false, the loop is never entered and no OSD is created as result. I hacked a workaround, which removes the last conditional check (occupied_slots < fast_slots_per_device) and hard-coded the expected size of each block.db in my deployment file: ``` service_type: osd service_id: delta2024_osd service_name: osd.delta2024_osd placement: label: delta2024 spec: block_db_size: 397G data_devices: rotational: 1 size: '15T:' db_devices: rotational: 0 encrypted: true filter_logic: AND objectstore: bluestore ``` This gives the expected results. In my opinion, cephadm sends a wrong „ceph-volume lvm batch“ command to the osd-node. It should always include all of the disks, since running it is promised to be idempotent. With the full list of disks, ceph-volume should be able to calculate correct slots for block.db. Did I find a bug here or is this expected behavior? --------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907 Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens

1 0

Re: Broken dashboard after upgrade to 20.2.0
by Ankush Behl 11 Mar '26

11 Mar '26

I assume this is happening on a cluster that is running on debian. Tracker: https://tracker.ceph.com/issues/74199 PR for Fix: https://github.com/ceph/ceph/pull/67133 We have raised a backport PR in the Tentacle branch to fix it. https://github.com/ceph/ceph/pull/67510 On Wed, Feb 25, 2026 at 12:25 PM Nicola Mori via ceph-users < ceph-users(a)ceph.io> wrote: > Any help? > > On 18/02/26 8:17 AM, Nicola Mori wrote: > > I found that the error is reproducible from Ceph CLI: > > > > # ceph dashboard get-prometheus-api-host > > Error EIO: Module 'dashboard' has experienced an error and cannot handle > > commands: No module named 'smb' > > > > Any hint? > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 2

An appeal for community help: mailing lists
by Patrick Donnelly 10 Mar '26

10 Mar '26

Hello, As you may have noticed, the ceph mailing lists have struggled with reliable delivery of mail postings. Additionally, emails sent to the lists are rewritten to comply with DMARC authenticity checks. This means you will get duplicates frequently. Unfortunately the Ceph project lacks modern expertise and time to make mailing lists work with the new rules pioneered by the email monopolists. We really could use some help from knowledgeable folks who would be willing to manage or advise on software (we are currently using GNU Mailman) and configurations that will work. Ideally, like LKML, the mailing list should be a dumb reflector as much as possible (no mail rewrites!). If you would be willing to contribute your time towards this, please get in touch with me privately off-list. Thanks, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: reef 18.2.8 QE validation status
by Patrick Donnelly 10 Mar '26

10 Mar '26

The `fs` and `powercycle` suites will need rerun with [1]. I believe you're already doing that but I'm documenting that for the list. I'm going to look into the upgrade issues today [2]. [1] https://github.com/ceph/ceph/pull/67495 [2] https://tracker.ceph.com/issues/75034 On Mon, Feb 23, 2026 at 4:29 PM Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> wrote: > > Patrick, > > The only outstanding issues are the FS approval and upgrade issues, > reported by Laura. > > What do you suggest we do next here? > > On Mon, Feb 23, 2026 at 8:54 AM Adam King via Dev <dev(a)ceph.io> wrote: > > > orch approved. Run looks ugly due to us no having a proper ignorelist for > > the orch suite on reef, but the actual results are okay > > > > On Sat, Nov 29, 2025 at 12:09 PM Yuri Weinstein via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > >> Rerun scheduled for krbd > >> > >> Still waiting for replies from dev leads > >> > >> On Mon, Nov 24, 2025 at 6:24 AM Ilya Dryomov <idryomov(a)gmail.com> wrote: > >> > >> > On Fri, Nov 21, 2025 at 12:04 AM Yuri Weinstein <yweinste(a)redhat.com> > >> > wrote: > >> > > > >> > > Note: this is an attempt to test and publish post EOL reef point > >> release > >> > > > >> > > Details of this release are summarized here: > >> > > > >> > > https://tracker.ceph.com/issues/73906#note-1 > >> <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> > >> > > Release Notes - TBD > >> > > LRC upgrade - N/A > >> > > > >> > > distro => jammy focal centos9 windows bookworm > >> > > > >> > > Seeking approvals/reviews for: > >> > > > >> > > smoke - Laura approved? > >> > > > >> > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > >> approved? > >> > > (rados is also running on build 2 == build 1 + > >> > > https://github.com/ceph/ceph/pull/66056 > >> <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Fgithub.com%2Fceph%2…> > >> per Igor's request) > >> > > > >> > > rgw - Adam E approved? > >> > > > >> > > fs - Venky approved? > >> > > > >> > > orch - Adam King approved? > >> > > > >> > > rbd - Ilya approved? > >> > > krbd - Ilya approved? > >> > > >> > rbd approved, krbd needs a rerun as all jobs failed while > >> > going through cephlab.yml playbook (not related to RBD): > >> > > >> > ['Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs type, > >> > bad option, bad superblock on /dev/mapper/vg_nvme-lv_5, missing > >> > codepage or helper program, or other error.'] > >> > > >> > Thanks, > >> > > >> > Ilya > >> > > >> > > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > Dev mailing list -- dev(a)ceph.io > > To unsubscribe send an email to dev-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 1

Can rbd sparsify be run on a running VM's disk?
by Steve Yates 10 Mar '26

10 Mar '26

Question: Can rbd sparsify safely be run on a running VM's disk, in Ceph 19/Squid? Or must the VM be shut down first? I tried looking online but am not finding a direct answer. More info: I'm looking into why "rbd du" shows usage inconsistent with "ceph df" or Proxmox's "usage" displays (Proxmox thread<https://forum.proxmox.com/threads/ceph-rbd-du-shows-usage-2-4x-higher-than-…>). Sparsify on a stopped VM reduces "rbd du" usage, and the number of objects, but doesn't change the amount of space used according to Ceph and PVE. I am wondering if sparsify can be safely scripted. Or if I should just ignore "rbd du" USED amounts. Thanks.

1 0

Re: tentacle 20.2.1 QE validation status
by Afreen 10 Mar '26

10 Mar '26

Dashboard approved. There is a known issue which is a flake being fixed here https://tracker.ceph.com/issues/70669 On Tue, Mar 10, 2026 at 12:17 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/74838#note-4 > > Release Notes - TBD > LRC upgrade - TBD > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya > krbd: Ilya (still running) > nvmeof: Aviv, Vallari > > upgrade/squid-x: Laura > upgrade/reef-x: Laura > upgrade/telemetry-upgrade: Laura > > upgrade/tentacle-p2p: N/A > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED > upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED > > perf-basic: PASSED > > crimson-rados: N/A > > powercycle: PASSED > > TIA > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0