Re: Can rbd sparsify be run on a running VM's disk?
Hi Alwin, Thanks for the input. Sounds like "stopped" is the safe way, which is what I'd assumed. It would be good if that was clarified in the Ceph docs (https://docs.ceph.com/en/latest/man/8/rbd/#:~:text=sparsify). I found in the Proxmox forum thread that if I create 1+10 GB in two zero files on a small VM, delete, and trim, "rbd du" USED remained at 37/43 GB for that VM, so that doesn't help. Also same issue on Windows VMs, and we have at least one (a pfSense VM) using ZFS, so it's not just EXT4. Re: the serverfault thread, rebooting a VM results in no change in "rbd du" USED. We do regularly reboot VMs for updates, and migrate too when rebooting PVE nodes. I'm about at the point of ignoring the "rbd du" USED number. It doesn't seem terribly meaningful. -----Original Message----- From: Alwin Antreich via ceph-users <ceph-users@ceph.io> Sent: Wednesday, March 11, 2026 6:27 AM To: Steve Yates <steve@teamits.com> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Can rbd sparsify be run on a running VM's disk? Hi Steve, I'd advise against using sparsify, especially on any mapped RBD (running VM). We had a support case once where this operation on a live VM caused an OSD lockup, halting the operation for other clients. Sparsify really tries to find 0-filled objects that can be removed. It's best to stick with fstrim (or equivalent) in the guest. rbd du estimates size cheaply by counting the number of objects and multiplying that count by the set object size. Therefore it diverges from the size you see of the guest's filesystem. Some additional points may contribute to the difference: * An OSD has a 4K (SSD) / 64K (HDD) threshold (bluestore_min_alloc_size) for discard operations, meaning any discard smaller than this threshold is ignored. * Disk encryption often doesn't clear block for security. * EXT4 uses an in-memory bitmap that does not trim blocks if they haven't been overwritten before. This safeguards against wear on consumer SSDs but has the disadvantage that a storage move will thick-allocate the disk and an fstrim isn't clearing it. The only option is to write a big file and delete it or reboot. See: https://forum.proxmox.com/threads/help-with-trim-on-virtio-scsi-single.12381... I hope this helps. Cheers, Alwin On Tue, 10 Mar 2026 at 14:59, Steve Yates via ceph-users <ceph-users@ceph.io> wrote:
Question: Can rbd sparsify safely be run on a running VM's disk, in Ceph 19/Squid? Or must the VM be shut down first? I tried looking online but am not finding a direct answer.
More info: I'm looking into why "rbd du" shows usage inconsistent with "ceph df" or Proxmox's "usage" displays (Proxmox thread< https://forum.proxmox.com/threads/ceph-rbd-du-shows-usage-2-4x-higher-than-inside-vm.181348/>). Sparsify on a stopped VM reduces "rbd du" usage, and the number of objects, but doesn't change the amount of space used according to Ceph and PVE. I am wondering if sparsify can be safely scripted. Or if I should just ignore "rbd du" USED amounts.
Thanks. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alwin Antreich Head of Training and Proxmox Services Want to meet: https://calendar.app.google/MuA2isCGnh8xBb657 croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> TOP 100 Innovator Award <https://croit.io/blog/croit-receives-top-100-seal> Winner by compamedia Technology Fast50 Award <https://croit.io/blog/deloitte-technology-fast-50-award> Winner by Deloitte _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io ------------------------------------ This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click the following link to report it (https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1zdGV...).
Hi Steve, On Wed, 11 Mar 2026 at 15:26, Steve Yates <steve@teamits.com> wrote:
Hi Alwin,
Thanks for the input. Sounds like "stopped" is the safe way, which is what I'd assumed. It would be good if that was clarified in the Ceph docs ( https://docs.ceph.com/en/latest/man/8/rbd/#:~:text=sparsify).
I found in the Proxmox forum thread that if I create 1+10 GB in two zero files on a small VM, delete, and trim, "rbd du" USED remained at 37/43 GB for that VM, so that doesn't help. Also same issue on Windows VMs, and we have at least one (a pfSense VM) using ZFS, so it's not just EXT4.
AFAIK, in Proxmox VE, QEMU converts zero-writes to discards because "detect-zeroes":"unmap" is set on the disk drives. This suggests that the disks are highly fragmented and the object count doesn't change significantly. A similar thing may apply to ZFS as well. Sparsify discards an object when it consists entirely of zeros, at the object level. But an fstrim might not trim the entire range of an object because it operates at the block level. This potentially leaves empty objects behind, which sparsify will pick up. You could try a storage migration, which might perform a similar function to sparsify and could be done online.
Re: the serverfault thread, rebooting a VM results in no change in "rbd du" USED. We do regularly reboot VMs for updates, and migrate too when rebooting PVE nodes.
Some thoughts about the regular reboot. You've likely thought about this, but eh. :) * Is the discard flag set on the disks? * Is guest-trim enabled for qemu-guest-agent in the VM's options tab after migration/cloning? * Is the "SSD emulation" flag set on the disk tab of the VM? Especially for Windows, as it will switch from defragmentation to TRIM (amongst other things).
I'm about at the point of ignoring the "rbd du" USED number. It doesn't seem terribly meaningful.
Yeah, it's an indicator but not really something to worry about unless you're heavily overprovisioning your pools. Cheers, Alwin -- Alwin Antreich Head of Training and Proxmox Services Want to meet: https://calendar.app.google/MuA2isCGnh8xBb657 croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> TOP 100 Innovator Award <https://croit.io/blog/croit-receives-top-100-seal> Winner by compamedia Technology Fast50 Award <https://croit.io/blog/deloitte-technology-fast-50-award> Winner by Deloitte
Hi Alwin, All VM disks are set as SSD with discard enabled. “Run guest-trim after a disk move or VM migration” was not checked. I didn’t realize that was in there! I don’t think that applies though? Per https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_qemu_agent : “With this enabled, Proxmox VE will issue a trim command to the guest after the following operations that have the potential to write out zeros to the storage: * moving a disk to another storage * live migrating a VM to another node with local storage” …and using Ceph the disk doesn’t move if we migrate a VM…? Plus trim is being run weekly. Maybe I could stop a VM and move its disk to local storage and back into Ceph again but that would take quite a lot of time and seems like little to no gain except maybe a lower object count. That section also has this note on ext4, matching previous discussions: “There is a caveat with ext4 on Linux, because it uses an in-memory optimization to avoid issuing duplicate TRIM requests. Since the guest doesn’t know about the change in the underlying storage, only the first guest-trim will run as expected. Subsequent ones, until the next reboot, will only consider parts of the filesystem that changed since then.” From: Alwin Antreich <alwin.antreich@croit.io> Sent: Thursday, March 12, 2026 12:53 PM To: Steve Yates <steve@teamits.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: Can rbd sparsify be run on a running VM's disk? Hi Steve, On Wed, 11 Mar 2026 at 15:26, Steve Yates <steve@teamits.com<mailto:steve@teamits.com>> wrote: Hi Alwin, Thanks for the input. Sounds like "stopped" is the safe way, which is what I'd assumed. It would be good if that was clarified in the Ceph docs (https://docs.ceph.com/en/latest/man/8/rbd/#:~:text=sparsify). I found in the Proxmox forum thread that if I create 1+10 GB in two zero files on a small VM, delete, and trim, "rbd du" USED remained at 37/43 GB for that VM, so that doesn't help. Also same issue on Windows VMs, and we have at least one (a pfSense VM) using ZFS, so it's not just EXT4. AFAIK, in Proxmox VE, QEMU converts zero-writes to discards because "detect-zeroes":"unmap" is set on the disk drives. This suggests that the disks are highly fragmented and the object count doesn't change significantly. A similar thing may apply to ZFS as well. Sparsify discards an object when it consists entirely of zeros, at the object level. But an fstrim might not trim the entire range of an object because it operates at the block level. This potentially leaves empty objects behind, which sparsify will pick up. You could try a storage migration, which might perform a similar function to sparsify and could be done online. Re: the serverfault thread, rebooting a VM results in no change in "rbd du" USED. We do regularly reboot VMs for updates, and migrate too when rebooting PVE nodes. Some thoughts about the regular reboot. You've likely thought about this, but eh. :) * Is the discard flag set on the disks? * Is guest-trim enabled for qemu-guest-agent in the VM's options tab after migration/cloning? * Is the "SSD emulation" flag set on the disk tab of the VM? Especially for Windows, as it will switch from defragmentation to TRIM (amongst other things). I'm about at the point of ignoring the "rbd du" USED number. It doesn't seem terribly meaningful. Yeah, it's an indicator but not really something to worry about unless you're heavily overprovisioning your pools. Cheers, Alwin -- Alwin Antreich Head of Training and Proxmox Services Want to meet: https://calendar.app.google/MuA2isCGnh8xBb657 croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web<https://croit.io/> | LinkedIn<http://linkedin.com/company/croit> | Youtube<https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter<https://twitter.com/croit_io> TOP 100 Innovator Award<https://croit.io/blog/croit-receives-top-100-seal> Winner by compamedia Technology Fast50 Award<https://croit.io/blog/deloitte-technology-fast-50-award> Winner by Deloitte ________________________________ This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click here<https://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1zdGV2ZUB0ZWFtaXRzLmNvbTt0cz0xNzczMzM4MDAwO3V1aWQ9NjlCMkZEOEU2RDE2MUQ1MjBBN0ZCNkY3REVFMkUzMUI7dG9rZW49NjgyNWUzNWIwNjA3N2U0ZjBlMDg5NWRlYmZmZmU2ZWY2OTAyY2Q5MTs%3D> to report it.
Hi Steve, On Thu, 12 Mar 2026 at 19:16, Steve Yates <steve@teamits.com> wrote:
Hi Alwin,
All VM disks are set as SSD with discard enabled.
“Run guest-trim after a disk move or VM migration” was not checked. I didn’t realize that was in there! I don’t think that applies though? Per https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_qemu_agent :
“With this enabled, Proxmox VE will issue a trim command to the guest after the following operations that have the potential to write out zeros to the storage:
- moving a disk to another storage - live migrating a VM to another node with local storage”
…and using Ceph the disk doesn’t move if we migrate a VM…? Plus trim is being run weekly. Maybe I could stop a VM and move its disk to local storage and back into Ceph again but that would take quite a lot of time and seems like little to no gain except maybe a lower object count.
That's what I meant as an alternative.
That section also has this note on ext4, matching previous discussions: “There is a caveat with ext4 on Linux, because it uses an in-memory optimization to avoid issuing duplicate TRIM requests. Since the guest doesn’t know about the change in the underlying storage, only the first guest-trim will run as expected. Subsequent ones, until the next reboot, will only consider parts of the filesystem that changed since then.”
Exactly that. Even if the in-memory bitmap disappears, the underlying misalignment issue will remain and affect any guest. Cheers, Alwin
participants (2)
-
Alwin Antreich
-
Steve Yates