- ceph-users - lists.ceph.io

Re: Can rbd sparsify be run on a running VM's disk?
by Steve Yates 13 Mar '26

13 Mar '26

And am I reading that right, that after a reboot of an ext4 server TRIM will work again just fine, once? That still seems unrelated to the core issue of the incorrect USED space estimate since that wouldn't apply to Windows, ZFS, etc. Thanks, Steve -----Original Message----- From: Alwin Antreich via ceph-users <ceph-users(a)ceph.io> Sent: Friday, March 13, 2026 8:52 AM To: Steve Yates <steve(a)teamits.com> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Can rbd sparsify be run on a running VM's disk? Hi Steve, On Thu, 12 Mar 2026 at 19:16, Steve Yates <steve(a)teamits.com> wrote: > Hi Alwin, > > > > All VM disks are set as SSD with discard enabled. > > > > “Run guest-trim after a disk move or VM migration” was not checked. I > didn’t realize that was in there! I don’t think that applies though? Per > https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_qemu_agent : > > “With this enabled, Proxmox VE will issue a trim command to the guest > after the following operations that have the potential to write out zeros > to the storage: > > - moving a disk to another storage > - live migrating a VM to another node with local storage” > > > > …and using Ceph the disk doesn’t move if we migrate a VM…? Plus trim is > being run weekly. Maybe I could stop a VM and move its disk to local > storage and back into Ceph again but that would take quite a lot of time > and seems like little to no gain except maybe a lower object count. > That's what I meant as an alternative. > > > That section also has this note on ext4, matching previous discussions: > “There is a caveat with ext4 on Linux, because it uses an in-memory > optimization to avoid issuing duplicate TRIM requests. Since the guest > doesn’t know about the change in the underlying storage, only the first > guest-trim will run as expected. Subsequent ones, until the next reboot, > will only consider parts of the filesystem that changed since then.” > Exactly that. Even if the in-memory bitmap disappears, the underlying misalignment issue will remain and affect any guest. Cheers, Alwin _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------------ This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click the following link to report it (https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1zdG…)

1 0

Re: Can rbd sparsify be run on a running VM's disk?
by Steve Yates 13 Mar '26

13 Mar '26

Hi Alwin, Thanks for the input. Sounds like "stopped" is the safe way, which is what I'd assumed. It would be good if that was clarified in the Ceph docs (https://docs.ceph.com/en/latest/man/8/rbd/#:~:text=sparsify) I found in the Proxmox forum thread that if I create 1+10 GB in two zero files on a small VM, delete, and trim, "rbd du" USED remained at 37/43 GB for that VM, so that doesn't help. Also same issue on Windows VMs, and we have at least one (a pfSense VM) using ZFS, so it's not just EXT4. Re: the serverfault thread, rebooting a VM results in no change in "rbd du" USED. We do regularly reboot VMs for updates, and migrate too when rebooting PVE nodes. I'm about at the point of ignoring the "rbd du" USED number. It doesn't seem terribly meaningful. -----Original Message----- From: Alwin Antreich via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, March 11, 2026 6:27 AM To: Steve Yates <steve(a)teamits.com> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Can rbd sparsify be run on a running VM's disk? Hi Steve, I'd advise against using sparsify, especially on any mapped RBD (running VM). We had a support case once where this operation on a live VM caused an OSD lockup, halting the operation for other clients. Sparsify really tries to find 0-filled objects that can be removed. It's best to stick with fstrim (or equivalent) in the guest. rbd du estimates size cheaply by counting the number of objects and multiplying that count by the set object size. Therefore it diverges from the size you see of the guest's filesystem. Some additional points may contribute to the difference: * An OSD has a 4K (SSD) / 64K (HDD) threshold (bluestore_min_alloc_size) for discard operations, meaning any discard smaller than this threshold is ignored. * Disk encryption often doesn't clear block for security. * EXT4 uses an in-memory bitmap that does not trim blocks if they haven't been overwritten before. This safeguards against wear on consumer SSDs but has the disadvantage that a storage move will thick-allocate the disk and an fstrim isn't clearing it. The only option is to write a big file and delete it or reboot. See: https://forum.proxmox.com/threads/help-with-trim-on-virtio-scsi-single.1238… I hope this helps. Cheers, Alwin On Tue, 10 Mar 2026 at 14:59, Steve Yates via ceph-users <ceph-users(a)ceph.io> wrote: > Question: Can rbd sparsify safely be run on a running VM's disk, in Ceph > 19/Squid? Or must the VM be shut down first? I tried looking online but > am not finding a direct answer. > > More info: I'm looking into why "rbd du" shows usage inconsistent with > "ceph df" or Proxmox's "usage" displays (Proxmox thread< > https://forum.proxmox.com/threads/ceph-rbd-du-shows-usage-2-4x-higher-than-…>). > Sparsify on a stopped VM reduces "rbd du" usage, and the number of objects, > but doesn't change the amount of space used according to Ceph and PVE. I > am wondering if sparsify can be safely scripted. Or if I should just > ignore "rbd du" USED amounts. > > Thanks. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Alwin Antreich Head of Training and Proxmox Services Want to meet: https://calendar.app.google/MuA2isCGnh8xBb657 croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> TOP 100 Innovator Award <https://croit.io/blog/croit-receives-top-100-seal> Winner by compamedia Technology Fast50 Award <https://croit.io/blog/deloitte-technology-fast-50-award> Winner by Deloitte _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------------ This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click the following link to report it (https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1zdG…)

2 3

Re: Ceph Reef (18.2.4) OSD fails to start with “Permission denied” after Ubuntu 20.04 -> 22.04 upgrade (cephadm)
by Jay See 13 Mar '26

13 Mar '26

Hi Martin, Thanks for the suggestion. I'm not entirely sure what changed during the upgrade. I have more nodes in the same cluster and another cluster with additional nodes where the `ceph` user and group IDs are `64045`, and those are working without any issues. However, on the nodes that were upgraded, the OSDs failed with the permission errors. After changing the `ceph` UID/GID to `167`, everything started working correctly. Once again thanks for your help. ~ Jay. On Fri, Mar 13, 2026 at 1:07 PM Martin Andersen via ceph-users < ceph-users(a)ceph.io> wrote: > > * Jay See via ceph-users <ceph-users(a)ceph.io> on Wed, 11 Mar 2026 at > 09:41: > > > Hi all, > > > > After upgrading a Ceph Reef (18.2.4) node from Ubuntu 20.04 (kernel > 5.4.0) > > to Ubuntu 22.04 (kernel 5.15.0-94), a cephadm-managed OSD fails to start > > with a “Permission denied” error. > > Hi Jay, > > > > > We had a similar issue when moving from 20.04 to 24.04 in a containerized > setup. The issue for us turned out to be that Focal inherited the > uids+gids from > the OS into the containers, whereas Noble did not. Worth checking if it is > the > same on Jammy as well. > > The logs will state that the osd mounts are chowned to ceph:ceph, > sometimes > > > stating the actual numerical uid+gid (167:167). However, this turned out > to be > > > misleading, and one had to manually set the uid+gids on the storage nodes > as > > > these were set to a higher range than what was expected by the containers, > thus > > > causing the “permission denied” error- > > > > > $ usermod -u 167 ceph ; groupmod -g 167 ceph ; reboot > > --Martin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- P *SAVE PAPER – Please do not print this e-mail unless absolutely necessary.*

1 0

Re: Ceph Reef (18.2.4) OSD fails to start with “Permission denied” after Ubuntu 20.04 -> 22.04 upgrade (cephadm)
by Martin Andersen 13 Mar '26

13 Mar '26

* Jay See via ceph-users <ceph-users(a)ceph.io> on Wed, 11 Mar 2026 at 09:41: > Hi all, > > After upgrading a Ceph Reef (18.2.4) node from Ubuntu 20.04 (kernel 5.4.0) > to Ubuntu 22.04 (kernel 5.15.0-94), a cephadm-managed OSD fails to start > with a “Permission denied” error. Hi Jay, We had a similar issue when moving from 20.04 to 24.04 in a containerized setup. The issue for us turned out to be that Focal inherited the uids+gids from the OS into the containers, whereas Noble did not. Worth checking if it is the same on Jammy as well. The logs will state that the osd mounts are chowned to ceph:ceph, sometimes stating the actual numerical uid+gid (167:167). However, this turned out to be misleading, and one had to manually set the uid+gids on the storage nodes as these were set to a higher range than what was expected by the containers, thus causing the “permission denied” error- $ usermod -u 167 ceph ; groupmod -g 167 ceph ; reboot --Martin

1 0

Windows driver topics for Monday Steering Committe Re: Ceph Windows driver still has active development?
by Martin Wege 13 Mar '26

13 Mar '26

On Fri, Mar 13, 2026 at 10:23 AM Alexandre Jardon via Dev <dev(a)ceph.io> wrote: > > There is a point on the monday agenda of the Steering Committee to see what will be the ceph posture on this Subject I have a scheduling conflict for that meeting, but I did some research which should be discussed in that call: There are three options to fix this: 1. Fix and improve the existing DOKAN-based driver 2. Implement a new RDBSS-based filesystem driver. This is what SMB, CIFS, RDP, WebDAV, NFSv3, NFSv4.1, BeeFS, VirtualBox SharedFolder drivers do 3. Implement a "raw" Windows driver without the aid of RDBSS. This is what OpenAFS is doing [1] is limited by the features of the Windows DOKAN kernel driver, which excludes Windows ACLs, sparse files Cygwin/MSYS2 support, and a purely synchronous interface. DOKAN and DOKAN-Y also saw almost zero development in the last couple of years. Advantages: It would be a minimum effort to do fixes there. [2] is the most common option for Windows network filesystems, as RDBSS is the kernel utility library for that. But the knowledge of writing a RDBSS driver is mostly limited to Microsoft staff, with the only exceptions being the VirtualBox Shared Folder driver, and the opensource Windows NFSv4.2 driver ms-nfs41-client. Advantages: RDBSS includes support for async IO, ACLs, use of local files as offline cache, sparse file support, Windows EA and Windows named stream support, Cygwin/MSYS2 support. This is a full time job, and requires hiring. [3] would be the most flexible option, because it can do anything a local filesystem driver can do, but likely the most expensive one, as no utility library is involved. All has to be done, and tested, from scratch. Maybe hire one of the OpenAFS guys to do that work. Will also be a full time job. Finally, a solution must be found to get the Windows drivers resulting from [1], [2], or [3] signed with a Secureboot certificate, otherwise the new and improved Ceph Windows driver can only be used with Secureboot disabled, which does not sit well with any enterprise IT setup. Thanks, Martin

1 0

Ceph Windows driver still has active development?
by Sebastian Feld 13 Mar '26

13 Mar '26

Is the Ceph Windows driver still actively being developed, or has the driver been abandoned? Looking at the repro and progress it seems nothing has happened since years: https://github.com/ketor/ceph-dokan Sebi -- Sebastian Feld - IT security consultant

1 0

Re: cephadm: Replacing OSD in mixed setup (block and block.db) is quite messy
by Alex Walender 11 Mar '26

11 Mar '26

Hey Anthony, The NVMEs are Micron 7500 Pro with 7.6TB space (MTFDKCC7T6TGP-1BK1DABYYR). We haven't exactly aimed for multipath disks. Lets say that they were ordered by „accident“ ;) Nevertheless, I was able to pinpoint the issue. It also has something to do with multipath-disks. The issue is described here https://tracker.ceph.com/issues/63862 TL;DR: The inventory command for ceph-volume was not able to list mpath-disks when they were already used as osd. They got falsely filtered out because they got detected as mapper devices with LVs on top. I added a patch-file for this https://tracker.ceph.com/attachments/8006 It appears that this only affects Ceph Reef and older. The code for ceph-volume/util/disk.py has been changed since then. --------------------------- M.Sc Alex Walender Institut für Bio- und Geowissenschaften IBG 5 - Computergestützte Metagenomik / de.NBI Cloud Site Bielefeld Büro : Universität Bielefeld (UHG), N7-101 Tel. : +49-521-106-2907 Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens > Am 11.03.2026 um 15:43 schrieb Anthony D'Atri via ceph-users <ceph-users(a)ceph.io>: > >> >> - 36 data disks, registered as mpath-devices >> - 2 NVMEs, which act as block.db for all 36 spinning disks. > > That’s an unusually high ratio. Which model exactly are the NVMe SSDs? And why are you using multipathing with the HDDs? > >> I hacked a workaround, which removes the last conditional check (occupied_slots < fast_slots_per_device) and hard-coded the expected size of each block.db in my deployment file: > > Honestly that’s what I often do, especially when applying the orch service to systems with OSDs that were previously deployed under a different scheme. Hybrid OSDs can be messy. > >> service_type: osd >> service_id: delta2024_osd >> service_name: osd.delta2024_osd >> placement: >> label: delta2024 >> spec: >> block_db_size: 397G >> data_devices: >> rotational: 1 >> size: '15T:' >> db_devices: >> rotational: 0 >> encrypted: true >> filter_logic: AND >> objectstore: bluestore >> >> This gives the expected results. >> In my opinion, cephadm sends a wrong „ceph-volume lvm batch“ command to the osd-node. It should always include all of the disks, since running it is promised to be idempotent. With the full list of disks, ceph-volume should be able to calculate correct slots for block.db. >> >> Did I find a bug here or is this expected behavior? > > Good question. Bitte enter a ticket at tracker.ceph.com <http://tracker.ceph.com/>. > >> >> >> >> --------------------------- >> M.Sc Alex Walender >> Institut für Bio- und Geowissenschaften >> IBG 5 - Computergestützte Metagenomik / >> de.NBI Cloud Site Bielefeld >> Büro : Universität Bielefeld (UHG), N7-101 >> Tel. : +49-521-106-2907 >> >> Forschungszentrum Jülich GmbH >> 52425 Jülich >> Sitz der Gesellschaft: Jülich >> Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 >> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >> Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

cephfs prone to race conditions?
by Carsten Götze 11 Mar '26

11 Mar '26

Hello! I'm having a problem with cephfs where clients can read past the end of files that have not yet been completely written to the storage. For example if I copy a large file to cephfs and immediately start to copy this large file to another file the second copy will eventually get truncated if the second process is faster than the first one. I have tested both the kernel-driver and the fuse-driver, both with sync and async option and both seem to be prone to the above behavior. Am I missing some mount option here, or is this a bug? With best regards Carsten Götze ------------------------------------------------------------------ Carsten Goetze Computer Graphics tel: +49 531 391-2109 TU Braunschweig fax: +49 531 391-2103 Muehlenpfordtstr. 23 eMail: c.goetze(a)cg.cs.tu-bs.de D-38106 Braunschweig http://www.cg.cs.tu-bs.de/people/goetze

1 0

Re: Can rbd sparsify be run on a running VM's disk?
by Steve Yates 11 Mar '26

11 Mar '26

If there is any potential danger and no functional gain from sparsify then it doesn't seem like a good idea to run it, and definitely not on a running VM. This cluster was created on 19.2.x. fast-diff and object-map should be on by default. "rbd du" takes a second or two to run. I've been looking at it for a week or so; the USED and OBJECTS change immediately after a sparsify but space used per PVE/"rbd df" doesn't functionally change other than the normal slight variance day to day, or a bit more one day or so each week which I assumed was TRIM. In other words, AFAICT, TRIM operates fine but doesn't change the "rbd du" USED output. Which by extension just increases over time, presumably up to the VM disk size. It will just look weird, eventually having more USED than cluster capacity, if anyone in the distant future runs that command (since PVE shows the actual used space). Currently bluestore_min_alloc_size = 4096 on all OSDs. bluestore_use_optimal_io_size_for_min_alloc_size defaults off, and says "It is a no-op for conventional media" but I can look into it. https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#c… > cool kids skate XFS "I used to be with 'it', but then they changed what 'it' was. Now what I'm with isn't 'it' anymore and what's 'it' seems weird and scary." 👴 -----Original Message----- From: Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, March 11, 2026 9:58 AM To: Steve Yates <steve(a)teamits.com> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: Can rbd sparsify be run on a running VM's disk? > >> I'm about at the point of ignoring the "rbd du" USED number. It doesn't seem terribly meaningful. > If all of your clients are the equivalent of Luminous or later, do you have the fast-diff and object-map feature flags enabled by default and for existing volumes? They dramatically increase the speed of `rbd du`. Also, ISTR that there may be an asynchronous GC process where the freed capacity takes some time to show up. Maybe run the du again and see if it’s changed? > Sparsify really tries to find 0-filled objects that can be removed. It's best to stick with > fstrim (or equivalent) in the guest. Agreed. Sparsify is best reserved for volumes that are currently unattached. > An OSD has a 4K (SSD) / 64K (HDD) threshold (bluestore_min_alloc_size) This used to default to 64 KiB; in the O/P era these values were split and varied a bit before both settling on 4 KiB. In recent releases they are both 4 KiB to reduce space amp, e.g. for tiny RGW objects. Note that despite some references on the net, you cannot effectively retrofit this value for an existing OSD. The process will report a runtime change, but it’s actually baked into the OSD and cannot be changed in practice without redeploying. From … Quincy I think it was, the baked-in value is reported by `ceph osd metadata`. Some outdated references suggest setting the value based on workload, I suggest not doing that unless you are using coarse-IU SSDs e.g. P4326, P5316, P5336, 6550, et al. I suggest setting bluestore_use_optimal_io_size_for_min_alloc_size to true before creating OSDs. I have never seen a case where it does the wrong thing. > EXT4 uses > All the cool kids skate XFS, warts and all ;) _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------------ This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click the following link to report it (https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1zdG…)

1 0

Re: Can rbd sparsify be run on a running VM's disk?
by Anthony D'Atri 11 Mar '26

11 Mar '26

> >> I'm about at the point of ignoring the "rbd du" USED number. It doesn't seem terribly meaningful. > If all of your clients are the equivalent of Luminous or later, do you have the fast-diff and object-map feature flags enabled by default and for existing volumes? They dramatically increase the speed of `rbd du`. Also, ISTR that there may be an asynchronous GC process where the freed capacity takes some time to show up. Maybe run the du again and see if it’s changed? > Sparsify really tries to find 0-filled objects that can be removed. It's best to stick with > fstrim (or equivalent) in the guest. Agreed. Sparsify is best reserved for volumes that are currently unattached. > An OSD has a 4K (SSD) / 64K (HDD) threshold (bluestore_min_alloc_size) This used to default to 64 KiB; in the O/P era these values were split and varied a bit before both settling on 4 KiB. In recent releases they are both 4 KiB to reduce space amp, e.g. for tiny RGW objects. Note that despite some references on the net, you cannot effectively retrofit this value for an existing OSD. The process will report a runtime change, but it’s actually baked into the OSD and cannot be changed in practice without redeploying. From … Quincy I think it was, the baked-in value is reported by `ceph osd metadata`. Some outdated references suggest setting the value based on workload, I suggest not doing that unless you are using coarse-IU SSDs e.g. P4326, P5316, P5336, 6550, et al. I suggest setting bluestore_use_optimal_io_size_for_min_alloc_size to true before creating OSDs. I have never seen a case where it does the wrong thing. > EXT4 uses > All the cool kids skate XFS, warts and all ;)

1 0