Orphaned CephFS objects
Hi, Some months ago we deleted about 1.4PB net of CephFS data. Approximately 600TB net of the space we expected to reclaim, did not get reclaimed. The cluster in question is now on Reef, but it was on Pacific when this happened. The cluster has 660 15TB NVMe OSD's on 55 nodes, plus 5 separate mon nodes. We almost exclusively use it for CephFS, though there is a small RBD pool for VM images. At the time we had a single MDS for the entire cluster, but we are now breaking that CephFS instance apart into multiple instances to spread the load (we tried multiple MDS ranks but it regressed performance and caused a near-disaster). Anyway, we often delete ~500TB at a time and we haven't previously run into this orphaned object problem. At least, not at a scale big enough for us to notice -- generally I've gotten back the space I expected to free up. Also we have a 30PB raw spinning disk cluster running Quincy that we deleted 2 PB net from at the same time, and it didn't exhibit this problem. So my assumption is that there is a bug in the version we were running that we exposed by deleting so much data at once. Our support vendor did some digging using some special utilities and found that there are a bunch of orphaned objects from 2 particular snapshots, where the MDS has no reference to the objects, but they still exist on disk. They feel that it would be somewhat difficult and risky to try to find and delete these objects via rados. However, obviously, 600+ *net* TB of all-NVMe storage is quite a lot of money to just let go to waste. That's effectively over 1.1 PB, once you figure EC overhead and the need to not fill the cluster over about 70%. So to reclaim the pool I have created a plan to move everything off of the pool, and then delete the pool. Which we are largely needing to do anyway, due to splitting to multiple CephFS instances. My question is, am I likely to run into problems with this? For example, will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk? Thanks a lot for any insight, Trey Palmer
Some months ago we deleted about 1.4PB net of CephFS data. Approximately 600TB net of the space we expected to reclaim, did not get reclaimed.
I always have to ask: any chance that there are `rados bench` orphans in there? I've encounter this a few times from runs that didn't properly clean up.
They feel that it would be somewhat difficult and risky to try to find and delete these objects via rados. However, obviously, 600+ *net* TB of all-NVMe storage is quite a lot of money to just let go to waste. That's effectively over 1.1 PB, once you figure EC overhead and the need to not fill the cluster over about 70%.
Why 70%?
So to reclaim the pool I have created a plan to move everything off of the pool, and then delete the pool. Which we are largely needing to do anyway, due to splitting to multiple CephFS instances.
My question is, am I likely to run into problems with this? For example, will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
There is design work for a future ability to migrate a pool transparently, for example to effect a new EC profile, but that won't be available anytime soon. Is the pool in question the first/default CephFS data pool, or one added?
On Thu, Oct 2, 2025 at 9:45 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
There is design work for a future ability to migrate a pool transparently, for example to effect a new EC profile, but that won't be available anytime soon.
This is, unfortunately, irrelevant in this case. Migrating a pool will migrate all the objects and their snapshots, even the unwanted ones. What Trey has (as far as I understood) is that there are some RADOS-level snapshots that do not correspond to any CephFS-level snapshots and are thus garbage, not to be migrated. That's why the talk about file migration and not pool-level operations. Now to the original question:
will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
Just tested in a test cluster - it won't balk and won't demand force even if you remove a pool that is actually used by files. So beware. $ ceph osd pool create badfs_evilpool 32 ssd-only pool 'badfs_evilpool' created $ ceph fs add_data_pool badfs badfs_evilpool added data pool 38 to fsmap $ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_wrongpool cephfs_data_rightpool cephfs_data_hdd ] name: badfs, metadata pool: badfs_metadata, data pools: [badfs_data badfs_evilpool ] $ cephfs-shell -f badfs CephFS:~/>>> ls dir1/ dir2/ CephFS:~/>>> mkdir evil CephFS:~/>>> setxattr evil ceph.dir.layout.pool badfs_evilpool ceph.dir.layout.pool is successfully set to badfs_evilpool CephFS:~/>>> put /usr/bin/ls /evil/ls $ ceph fs rm_data_pool badfs badfs_evilpool removed data pool 38 from fsmap -- Alexander Patrakov
Hi, Here user that suffer years ago a problem with orphans. Years ago, after much research, we discovered that for some reason the WALLDB/Metadata entries were being deleted and corrupted, but the data on the disks weren't physically erased. Sometimes the garbage collector (deferred delete) would fail and skip the deletion, leaving hundreds of TB behind. Speaking with other heavy CEPH users, they were aware of this and couldn't find a great solution either, just they instead use replica 3 , used replica 4. (big customers with big budget) At the time, we were presented with two options: wipe each disk, and CEPH would only rebuild the data it knows is valid, but that would take time, maybe in your case where your full NVME will take not too much. Or create a new cluster and move the valid data. In our case ceph orphan tool start looping due bugs and didn’t provide a real solution, our case 1PB ceph, with aprox 300TB orphaned. I remember ceph orphan tool running for weeks ☹ bad ass time. Our ceph use case : S3 and version 12 to 14 nautilus... Sometimes , we as administrators doesn’t care about this issues until you need to wipe a lot of data. And you use simple calc and don’t match. Regards, -----Mensaje original----- De: Alexander Patrakov <patrakov@gmail.com> Enviado el: jueves, 2 de octubre de 2025 22:56 Para: Anthony D'Atri <anthony.datri@gmail.com> CC: ceph-users@ceph.io Asunto: [ceph-users] Re: Orphaned CephFS objects On Thu, Oct 2, 2025 at 9:45 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
There is design work for a future ability to migrate a pool transparently, for example to effect a new EC profile, but that won't be available anytime soon.
This is, unfortunately, irrelevant in this case. Migrating a pool will migrate all the objects and their snapshots, even the unwanted ones. What Trey has (as far as I understood) is that there are some RADOS-level snapshots that do not correspond to any CephFS-level snapshots and are thus garbage, not to be migrated. That's why the talk about file migration and not pool-level operations. Now to the original question:
will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
Just tested in a test cluster - it won't balk and won't demand force even if you remove a pool that is actually used by files. So beware. $ ceph osd pool create badfs_evilpool 32 ssd-only pool 'badfs_evilpool' created $ ceph fs add_data_pool badfs badfs_evilpool added data pool 38 to fsmap $ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_wrongpool cephfs_data_rightpool cephfs_data_hdd ] name: badfs, metadata pool: badfs_metadata, data pools: [badfs_data badfs_evilpool ] $ cephfs-shell -f badfs CephFS:~/>>> ls dir1/ dir2/ CephFS:~/>>> mkdir evil CephFS:~/>>> setxattr evil ceph.dir.layout.pool badfs_evilpool ceph.dir.layout.pool is successfully set to badfs_evilpool CephFS:~/>>> put /usr/bin/ls /evil/ls $ ceph fs rm_data_pool badfs badfs_evilpool removed data pool 38 from fsmap -- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
FWIW, my understanding is that this RGW issue was fixed several releases ago. The OP’s cluster IIRC is mostly CephFS, so I suspect something else is going on.
On Oct 2, 2025, at 7:29 PM, Manuel Rios - EDH <mriosfer@easydatahost.com> wrote:
Hi,
Here user that suffer years ago a problem with orphans.
Years ago, after much research, we discovered that for some reason the WALLDB/Metadata entries were being deleted and corrupted, but the data on the disks weren't physically erased. Sometimes the garbage collector (deferred delete) would fail and skip the deletion, leaving hundreds of TB behind. Speaking with other heavy CEPH users, they were aware of this and couldn't find a great solution either, just they instead use replica 3 , used replica 4. (big customers with big budget) At the time, we were presented with two options: wipe each disk, and CEPH would only rebuild the data it knows is valid, but that would take time, maybe in your case where your full NVME will take not too much. Or create a new cluster and move the valid data.
In our case ceph orphan tool start looping due bugs and didn’t provide a real solution, our case 1PB ceph, with aprox 300TB orphaned.
I remember ceph orphan tool running for weeks ☹ bad ass time.
Our ceph use case : S3 and version 12 to 14 nautilus...
Sometimes , we as administrators doesn’t care about this issues until you need to wipe a lot of data. And you use simple calc and don’t match.
Regards,
-----Mensaje original----- De: Alexander Patrakov <patrakov@gmail.com> Enviado el: jueves, 2 de octubre de 2025 22:56 Para: Anthony D'Atri <anthony.datri@gmail.com> CC: ceph-users@ceph.io Asunto: [ceph-users] Re: Orphaned CephFS objects
On Thu, Oct 2, 2025 at 9:45 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
There is design work for a future ability to migrate a pool transparently, for example to effect a new EC profile, but that won't be available anytime soon.
This is, unfortunately, irrelevant in this case. Migrating a pool will migrate all the objects and their snapshots, even the unwanted ones. What Trey has (as far as I understood) is that there are some RADOS-level snapshots that do not correspond to any CephFS-level snapshots and are thus garbage, not to be migrated.
That's why the talk about file migration and not pool-level operations.
Now to the original question:
will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
Just tested in a test cluster - it won't balk and won't demand force even if you remove a pool that is actually used by files. So beware.
$ ceph osd pool create badfs_evilpool 32 ssd-only pool 'badfs_evilpool' created $ ceph fs add_data_pool badfs badfs_evilpool added data pool 38 to fsmap $ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_wrongpool cephfs_data_rightpool cephfs_data_hdd ] name: badfs, metadata pool: badfs_metadata, data pools: [badfs_data badfs_evilpool ] $ cephfs-shell -f badfs CephFS:~/>>> ls dir1/ dir2/ CephFS:~/>>> mkdir evil CephFS:~/>>> setxattr evil ceph.dir.layout.pool badfs_evilpool ceph.dir.layout.pool is successfully set to badfs_evilpool CephFS:~/>>> put /usr/bin/ls /evil/ls $ ceph fs rm_data_pool badfs badfs_evilpool removed data pool 38 from fsmap
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
There was a RADOS bug with trimming deleted snapshots, but I don’t have the tracker ticket for more details — it involved some subtleties with one of the snaptrim reworks. Since CephFS doesn’t have any link to the snapshots, I suspect it was that. -Greg On Thu, Oct 2, 2025 at 5:08 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
FWIW, my understanding is that this RGW issue was fixed several releases ago. The OP’s cluster IIRC is mostly CephFS, so I suspect something else is going on.
On Oct 2, 2025, at 7:29 PM, Manuel Rios - EDH <mriosfer@easydatahost.com> wrote:
Hi,
Here user that suffer years ago a problem with orphans.
Years ago, after much research, we discovered that for some reason the WALLDB/Metadata entries were being deleted and corrupted, but the data on the disks weren't physically erased. Sometimes the garbage collector (deferred delete) would fail and skip the deletion, leaving hundreds of TB behind. Speaking with other heavy CEPH users, they were aware of this and couldn't find a great solution either, just they instead use replica 3 , used replica 4. (big customers with big budget) At the time, we were presented with two options: wipe each disk, and CEPH would only rebuild the data it knows is valid, but that would take time, maybe in your case where your full NVME will take not too much. Or create a new cluster and move the valid data.
In our case ceph orphan tool start looping due bugs and didn’t provide a real solution, our case 1PB ceph, with aprox 300TB orphaned.
I remember ceph orphan tool running for weeks ☹ bad ass time.
Our ceph use case : S3 and version 12 to 14 nautilus...
Sometimes , we as administrators doesn’t care about this issues until you need to wipe a lot of data. And you use simple calc and don’t match.
Regards,
-----Mensaje original----- De: Alexander Patrakov <patrakov@gmail.com> Enviado el: jueves, 2 de octubre de 2025 22:56 Para: Anthony D'Atri <anthony.datri@gmail.com> CC: ceph-users@ceph.io Asunto: [ceph-users] Re: Orphaned CephFS objects
On Thu, Oct 2, 2025 at 9:45 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
There is design work for a future ability to migrate a pool transparently, for example to effect a new EC profile, but that won't be available anytime soon.
This is, unfortunately, irrelevant in this case. Migrating a pool will migrate all the objects and their snapshots, even the unwanted ones. What Trey has (as far as I understood) is that there are some RADOS-level snapshots that do not correspond to any CephFS-level snapshots and are thus garbage, not to be migrated.
That's why the talk about file migration and not pool-level operations.
Now to the original question:
will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
Just tested in a test cluster - it won't balk and won't demand force even if you remove a pool that is actually used by files. So beware.
$ ceph osd pool create badfs_evilpool 32 ssd-only pool 'badfs_evilpool' created $ ceph fs add_data_pool badfs badfs_evilpool added data pool 38 to fsmap $ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_wrongpool cephfs_data_rightpool cephfs_data_hdd ] name: badfs, metadata pool: badfs_metadata, data pools: [badfs_data badfs_evilpool ] $ cephfs-shell -f badfs CephFS:~/>>> ls dir1/ dir2/ CephFS:~/>>> mkdir evil CephFS:~/>>> setxattr evil ceph.dir.layout.pool badfs_evilpool ceph.dir.layout.pool is successfully set to badfs_evilpool CephFS:~/>>> put /usr/bin/ls /evil/ls $ ceph fs rm_data_pool badfs badfs_evilpool removed data pool 38 from fsmap
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
There was a RADOS bug with trimming deleted snapshots, but I don’t have the tracker ticket for more details — it involved some subtleties with one of the snaptrim reworks. Since CephFS doesn’t have any link to the snapshots, I suspect it was that. -Greg
This sounds like exactly what it was. And we deleted even more on a cluster on a newer version, and it didn't have this problem. It doesn't *completely* make sense to me that CephFS has no link to the snapshots, given that you access them via directories, and create and remove via mkdir and rmdir, which I would think surely must mean the MDS is aware of them? But I claim no sophisticated knowledge. I'm fairly new to CephFS, most of my Ceph experience (going back almost a decade) is with RadosGW. Thanks, Trey On Thu, Oct 2, 2025 at 11:28 PM Gregory Farnum <gfarnum@redhat.com> wrote:
There was a RADOS bug with trimming deleted snapshots, but I don’t have the tracker ticket for more details — it involved some subtleties with one of the snaptrim reworks. Since CephFS doesn’t have any link to the snapshots, I suspect it was that. -Greg
On Thu, Oct 2, 2025 at 5:08 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
FWIW, my understanding is that this RGW issue was fixed several releases ago. The OP’s cluster IIRC is mostly CephFS, so I suspect something else is going on.
On Oct 2, 2025, at 7:29 PM, Manuel Rios - EDH < mriosfer@easydatahost.com> wrote:
Hi,
Here user that suffer years ago a problem with orphans.
Years ago, after much research, we discovered that for some reason the WALLDB/Metadata entries were being deleted and corrupted, but the data on the disks weren't physically erased. Sometimes the garbage collector (deferred delete) would fail and skip the deletion, leaving hundreds of TB behind. Speaking with other heavy CEPH users, they were aware of this and couldn't find a great solution either, just they instead use replica 3 , used replica 4. (big customers with big budget) At the time, we were presented with two options: wipe each disk, and CEPH would only rebuild the data it knows is valid, but that would take time, maybe in your case where your full NVME will take not too much. Or create a new cluster and move the valid data.
In our case ceph orphan tool start looping due bugs and didn’t provide a real solution, our case 1PB ceph, with aprox 300TB orphaned.
I remember ceph orphan tool running for weeks ☹ bad ass time.
Our ceph use case : S3 and version 12 to 14 nautilus...
Sometimes , we as administrators doesn’t care about this issues until you need to wipe a lot of data. And you use simple calc and don’t match.
Regards,
-----Mensaje original----- De: Alexander Patrakov <patrakov@gmail.com> Enviado el: jueves, 2 de octubre de 2025 22:56 Para: Anthony D'Atri <anthony.datri@gmail.com> CC: ceph-users@ceph.io Asunto: [ceph-users] Re: Orphaned CephFS objects
On Thu, Oct 2, 2025 at 9:45 PM Anthony D'Atri <anthony.datri@gmail.com
wrote:
There is design work for a future ability to migrate a pool
transparently, for example to effect a new EC profile, but that won't be available anytime soon.
This is, unfortunately, irrelevant in this case. Migrating a pool will migrate all the objects and their snapshots, even the unwanted ones. What Trey has (as far as I understood) is that there are some RADOS-level snapshots that do not correspond to any CephFS-level snapshots and are thus garbage, not to be migrated.
That's why the talk about file migration and not pool-level operations.
Now to the original question:
will I be able to do 'ceph fs rm_data_pool' once there are no longer
any
objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
Just tested in a test cluster - it won't balk and won't demand force even if you remove a pool that is actually used by files. So beware.
$ ceph osd pool create badfs_evilpool 32 ssd-only pool 'badfs_evilpool' created $ ceph fs add_data_pool badfs badfs_evilpool added data pool 38 to fsmap $ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_wrongpool cephfs_data_rightpool cephfs_data_hdd ] name: badfs, metadata pool: badfs_metadata, data pools: [badfs_data badfs_evilpool ] $ cephfs-shell -f badfs CephFS:~/>>> ls dir1/ dir2/ CephFS:~/>>> mkdir evil CephFS:~/>>> setxattr evil ceph.dir.layout.pool badfs_evilpool ceph.dir.layout.pool is successfully set to badfs_evilpool CephFS:~/>>> put /usr/bin/ls /evil/ls $ ceph fs rm_data_pool badfs badfs_evilpool removed data pool 38 from fsmap
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Fri, Oct 3, 2025 at 8:26 AM Trey Palmer <nerdmagicatl@gmail.com> wrote:
There was a RADOS bug with trimming deleted snapshots, but I don’t have the tracker ticket for more details — it involved some subtleties with one of the snaptrim reworks. Since CephFS doesn’t have any link to the snapshots, I suspect it was that. -Greg
This sounds like exactly what it was. And we deleted even more on a cluster on a newer version, and it didn't have this problem.
It doesn't *completely* make sense to me that CephFS has no link to the snapshots, given that you access them via directories, and create and remove via mkdir and rmdir, which I would think surely must mean the MDS is aware of them?
The MDS/client created the snapshotted data, but deleting a snapshot is just a metadata update: it tells the monitor “please delete snapshot ID 1234”, and the monitor puts 1234 in the osdmap as a snapshot to be deleted. Then the OSDs do that work asynchronously and outside of cephfs’ supervision. So, that must have happened, and there was this bug in RADOS that meant the snapshot didn’t actually get (fully) trimmed. -Greg
But I claim no sophisticated knowledge. I'm fairly new to CephFS, most of my Ceph experience (going back almost a decade) is with RadosGW.
Thanks,
Trey
On Thu, Oct 2, 2025 at 11:28 PM Gregory Farnum <gfarnum@redhat.com> wrote:
There was a RADOS bug with trimming deleted snapshots, but I don’t have the tracker ticket for more details — it involved some subtleties with one of the snaptrim reworks. Since CephFS doesn’t have any link to the snapshots, I suspect it was that. -Greg
On Thu, Oct 2, 2025 at 5:08 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
FWIW, my understanding is that this RGW issue was fixed several releases ago. The OP’s cluster IIRC is mostly CephFS, so I suspect something else is going on.
On Oct 2, 2025, at 7:29 PM, Manuel Rios - EDH < mriosfer@easydatahost.com> wrote:
Hi,
Here user that suffer years ago a problem with orphans.
Years ago, after much research, we discovered that for some reason the WALLDB/Metadata entries were being deleted and corrupted, but the data on the disks weren't physically erased. Sometimes the garbage collector (deferred delete) would fail and skip the deletion, leaving hundreds of TB behind. Speaking with other heavy CEPH users, they were aware of this and couldn't find a great solution either, just they instead use replica 3 , used replica 4. (big customers with big budget) At the time, we were presented with two options: wipe each disk, and CEPH would only rebuild the data it knows is valid, but that would take time, maybe in your case where your full NVME will take not too much. Or create a new cluster and move the valid data.
In our case ceph orphan tool start looping due bugs and didn’t provide a real solution, our case 1PB ceph, with aprox 300TB orphaned.
I remember ceph orphan tool running for weeks ☹ bad ass time.
Our ceph use case : S3 and version 12 to 14 nautilus...
Sometimes , we as administrators doesn’t care about this issues until you need to wipe a lot of data. And you use simple calc and don’t match.
Regards,
-----Mensaje original----- De: Alexander Patrakov <patrakov@gmail.com> Enviado el: jueves, 2 de octubre de 2025 22:56 Para: Anthony D'Atri <anthony.datri@gmail.com> CC: ceph-users@ceph.io Asunto: [ceph-users] Re: Orphaned CephFS objects
On Thu, Oct 2, 2025 at 9:45 PM Anthony D'Atri < anthony.datri@gmail.com> wrote:
There is design work for a future ability to migrate a pool transparently, for example to effect a new EC profile, but that won't be available anytime soon.
This is, unfortunately, irrelevant in this case. Migrating a pool will migrate all the objects and their snapshots, even the unwanted ones. What Trey has (as far as I understood) is that there are some RADOS-level snapshots that do not correspond to any CephFS-level snapshots and are thus garbage, not to be migrated.
That's why the talk about file migration and not pool-level operations.
Now to the original question:
will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
Just tested in a test cluster - it won't balk and won't demand force even if you remove a pool that is actually used by files. So beware.
$ ceph osd pool create badfs_evilpool 32 ssd-only pool 'badfs_evilpool' created $ ceph fs add_data_pool badfs badfs_evilpool added data pool 38 to fsmap $ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_wrongpool cephfs_data_rightpool cephfs_data_hdd ] name: badfs, metadata pool: badfs_metadata, data pools: [badfs_data badfs_evilpool ] $ cephfs-shell -f badfs CephFS:~/>>> ls dir1/ dir2/ CephFS:~/>>> mkdir evil CephFS:~/>>> setxattr evil ceph.dir.layout.pool badfs_evilpool ceph.dir.layout.pool is successfully set to badfs_evilpool CephFS:~/>>> put /usr/bin/ls /evil/ls $ ceph fs rm_data_pool badfs badfs_evilpool removed data pool 38 from fsmap
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The MDS/client created the snapshotted data, but deleting a snapshot is just a metadata update: it tells the monitor “please delete snapshot ID 1234”, and the monitor puts 1234 in the osdmap as a snapshot to be deleted. Then the OSDs do that work asynchronously and outside of cephfs’ supervision. So, that must have happened, and there was this bug in RADOS that meant the snapshot didn’t actually get (fully) trimmed.
So, effectively the MDS hands it off to the mon and then deletes any reference to the snapshot from its own metadata, and then due to the bug the mon dropped the ball? That makes sense, if so. Thanks for the explanation. -- Trey On Fri, Oct 3, 2025 at 11:40 AM Gregory Farnum <gfarnum@redhat.com> wrote:
On Fri, Oct 3, 2025 at 8:26 AM Trey Palmer <nerdmagicatl@gmail.com> wrote:
There was a RADOS bug with trimming deleted snapshots, but I don’t have the tracker ticket for more details — it involved some subtleties with one of the snaptrim reworks. Since CephFS doesn’t have any link to the snapshots, I suspect it was that. -Greg
This sounds like exactly what it was. And we deleted even more on a cluster on a newer version, and it didn't have this problem.
It doesn't *completely* make sense to me that CephFS has no link to the snapshots, given that you access them via directories, and create and remove via mkdir and rmdir, which I would think surely must mean the MDS is aware of them?
The MDS/client created the snapshotted data, but deleting a snapshot is just a metadata update: it tells the monitor “please delete snapshot ID 1234”, and the monitor puts 1234 in the osdmap as a snapshot to be deleted. Then the OSDs do that work asynchronously and outside of cephfs’ supervision. So, that must have happened, and there was this bug in RADOS that meant the snapshot didn’t actually get (fully) trimmed. -Greg
But I claim no sophisticated knowledge. I'm fairly new to CephFS, most of my Ceph experience (going back almost a decade) is with RadosGW.
Thanks,
Trey
On Thu, Oct 2, 2025 at 11:28 PM Gregory Farnum <gfarnum@redhat.com> wrote:
There was a RADOS bug with trimming deleted snapshots, but I don’t have the tracker ticket for more details — it involved some subtleties with one of the snaptrim reworks. Since CephFS doesn’t have any link to the snapshots, I suspect it was that. -Greg
On Thu, Oct 2, 2025 at 5:08 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
FWIW, my understanding is that this RGW issue was fixed several releases ago. The OP’s cluster IIRC is mostly CephFS, so I suspect something else is going on.
On Oct 2, 2025, at 7:29 PM, Manuel Rios - EDH < mriosfer@easydatahost.com> wrote:
Hi,
Here user that suffer years ago a problem with orphans.
Years ago, after much research, we discovered that for some reason the WALLDB/Metadata entries were being deleted and corrupted, but the data on the disks weren't physically erased. Sometimes the garbage collector (deferred delete) would fail and skip the deletion, leaving hundreds of TB behind. Speaking with other heavy CEPH users, they were aware of this and couldn't find a great solution either, just they instead use replica 3 , used replica 4. (big customers with big budget) At the time, we were presented with two options: wipe each disk, and CEPH would only rebuild the data it knows is valid, but that would take time, maybe in your case where your full NVME will take not too much. Or create a new cluster and move the valid data.
In our case ceph orphan tool start looping due bugs and didn’t provide a real solution, our case 1PB ceph, with aprox 300TB orphaned.
I remember ceph orphan tool running for weeks ☹ bad ass time.
Our ceph use case : S3 and version 12 to 14 nautilus...
Sometimes , we as administrators doesn’t care about this issues until you need to wipe a lot of data. And you use simple calc and don’t match.
Regards,
-----Mensaje original----- De: Alexander Patrakov <patrakov@gmail.com> Enviado el: jueves, 2 de octubre de 2025 22:56 Para: Anthony D'Atri <anthony.datri@gmail.com> CC: ceph-users@ceph.io Asunto: [ceph-users] Re: Orphaned CephFS objects
On Thu, Oct 2, 2025 at 9:45 PM Anthony D'Atri < anthony.datri@gmail.com> wrote:
There is design work for a future ability to migrate a pool transparently, for example to effect a new EC profile, but that won't be available anytime soon.
This is, unfortunately, irrelevant in this case. Migrating a pool will migrate all the objects and their snapshots, even the unwanted ones. What Trey has (as far as I understood) is that there are some RADOS-level snapshots that do not correspond to any CephFS-level snapshots and are thus garbage, not to be migrated.
That's why the talk about file migration and not pool-level operations.
Now to the original question:
will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
Just tested in a test cluster - it won't balk and won't demand force even if you remove a pool that is actually used by files. So beware.
$ ceph osd pool create badfs_evilpool 32 ssd-only pool 'badfs_evilpool' created $ ceph fs add_data_pool badfs badfs_evilpool added data pool 38 to fsmap $ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_wrongpool cephfs_data_rightpool cephfs_data_hdd ] name: badfs, metadata pool: badfs_metadata, data pools: [badfs_data badfs_evilpool ] $ cephfs-shell -f badfs CephFS:~/>>> ls dir1/ dir2/ CephFS:~/>>> mkdir evil CephFS:~/>>> setxattr evil ceph.dir.layout.pool badfs_evilpool ceph.dir.layout.pool is successfully set to badfs_evilpool CephFS:~/>>> put /usr/bin/ls /evil/ls $ ceph fs rm_data_pool badfs badfs_evilpool removed data pool 38 from fsmap
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, Greg. Suggestions for remediation?
The MDS/client created the snapshotted data, but deleting a snapshot is just a metadata update: it tells the monitor “please delete snapshot ID 1234”, and the monitor puts 1234 in the osdmap as a snapshot to be deleted. Then the OSDs do that work asynchronously and outside of cephfs’ supervision. So, that must have happened, and there was this bug in RADOS that meant the snapshot didn’t actually get (fully) trimmed. -Greg
Hi Greg, This one? https://tracker.ceph.com/issues/64646 Symptoms: - CLONES are reported on 'rados df' while pool has no snapshots. - 'rados lssnap -p <pool_name>' command shows no snapshots but some clones are listed by 'rados listsnaps -p <pool_name> <object_name>' even sometimes with no 'head' object. @Trey, if this is the one --- make sure it is before running the command --- running a 'ceph osd pool force-remove-snap <pool_name>' should put all leaked clone objects back in the trim queue and the OSDs should get rid of them. Regards, Frédéric. Frédéric Nass Senior Ceph Engineer Ceph Ambassador, France +49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751> frederic.nass@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Le ven. 3 oct. 2025 à 05:19, Gregory Farnum <gfarnum@redhat.com> a écrit :
There was a RADOS bug with trimming deleted snapshots, but I don’t have the tracker ticket for more details — it involved some subtleties with one of the snaptrim reworks. Since CephFS doesn’t have any link to the snapshots, I suspect it was that. -Greg
On Thu, Oct 2, 2025 at 5:08 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
FWIW, my understanding is that this RGW issue was fixed several releases ago. The OP’s cluster IIRC is mostly CephFS, so I suspect something else is going on.
On Oct 2, 2025, at 7:29 PM, Manuel Rios - EDH < mriosfer@easydatahost.com> wrote:
Hi,
Here user that suffer years ago a problem with orphans.
Years ago, after much research, we discovered that for some reason the WALLDB/Metadata entries were being deleted and corrupted, but the data on the disks weren't physically erased. Sometimes the garbage collector (deferred delete) would fail and skip the deletion, leaving hundreds of TB behind. Speaking with other heavy CEPH users, they were aware of this and couldn't find a great solution either, just they instead use replica 3 , used replica 4. (big customers with big budget) At the time, we were presented with two options: wipe each disk, and CEPH would only rebuild the data it knows is valid, but that would take time, maybe in your case where your full NVME will take not too much. Or create a new cluster and move the valid data.
In our case ceph orphan tool start looping due bugs and didn’t provide a real solution, our case 1PB ceph, with aprox 300TB orphaned.
I remember ceph orphan tool running for weeks ☹ bad ass time.
Our ceph use case : S3 and version 12 to 14 nautilus...
Sometimes , we as administrators doesn’t care about this issues until you need to wipe a lot of data. And you use simple calc and don’t match.
Regards,
-----Mensaje original----- De: Alexander Patrakov <patrakov@gmail.com> Enviado el: jueves, 2 de octubre de 2025 22:56 Para: Anthony D'Atri <anthony.datri@gmail.com> CC: ceph-users@ceph.io Asunto: [ceph-users] Re: Orphaned CephFS objects
On Thu, Oct 2, 2025 at 9:45 PM Anthony D'Atri <anthony.datri@gmail.com
wrote:
There is design work for a future ability to migrate a pool
transparently, for example to effect a new EC profile, but that won't be available anytime soon.
This is, unfortunately, irrelevant in this case. Migrating a pool will migrate all the objects and their snapshots, even the unwanted ones. What Trey has (as far as I understood) is that there are some RADOS-level snapshots that do not correspond to any CephFS-level snapshots and are thus garbage, not to be migrated.
That's why the talk about file migration and not pool-level operations.
Now to the original question:
will I be able to do 'ceph fs rm_data_pool' once there are no longer
any
objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
Just tested in a test cluster - it won't balk and won't demand force even if you remove a pool that is actually used by files. So beware.
$ ceph osd pool create badfs_evilpool 32 ssd-only pool 'badfs_evilpool' created $ ceph fs add_data_pool badfs badfs_evilpool added data pool 38 to fsmap $ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_wrongpool cephfs_data_rightpool cephfs_data_hdd ] name: badfs, metadata pool: badfs_metadata, data pools: [badfs_data badfs_evilpool ] $ cephfs-shell -f badfs CephFS:~/>>> ls dir1/ dir2/ CephFS:~/>>> mkdir evil CephFS:~/>>> setxattr evil ceph.dir.layout.pool badfs_evilpool ceph.dir.layout.pool is successfully set to badfs_evilpool CephFS:~/>>> put /usr/bin/ls /evil/ls $ ceph fs rm_data_pool badfs badfs_evilpool removed data pool 38 from fsmap
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Frédéric, Thanks so much for looking into this. The documentation isn't all that clear, but my impression has been that pool snapshots are an entirely different thing from CephFS snapshots. At least the documentation says this, and it sounds from the bug report you posted like it's dealing with mon-managed snapshots? To avoid snap id collision between mon-managed snapshots and file system snapshots, pools with mon-managed snapshots are not allowed to be attached to a file system. Also, mon-managed snapshots can’t be created in pools already attached to a file system either. I'd love for my impression to be incorrect and to be able to fix it this way, though! Thanks again, Trey On Mon, Oct 6, 2025 at 10:26 AM Frédéric Nass <frederic.nass@clyso.com> wrote:
Hi Greg,
This one? https://tracker.ceph.com/issues/64646
Symptoms: - CLONES are reported on 'rados df' while pool has no snapshots. - 'rados lssnap -p <pool_name>' command shows no snapshots but some clones are listed by 'rados listsnaps -p <pool_name> <object_name>' even sometimes with no 'head' object.
@Trey, if this is the one --- make sure it is before running the command --- running a 'ceph osd pool force-remove-snap <pool_name>' should put all leaked clone objects back in the trim queue and the OSDs should get rid of them.
Regards, Frédéric.
Frédéric Nass
Senior Ceph Engineer
Ceph Ambassador, France
+49 89 215252-751 <https://call.ctrlq.org/+49%2089%20215252-751>
frederic.nass@clyso.com
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Le ven. 3 oct. 2025 à 05:19, Gregory Farnum <gfarnum@redhat.com> a écrit :
There was a RADOS bug with trimming deleted snapshots, but I don’t have the tracker ticket for more details — it involved some subtleties with one of the snaptrim reworks. Since CephFS doesn’t have any link to the snapshots, I suspect it was that. -Greg
On Thu, Oct 2, 2025 at 5:08 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
FWIW, my understanding is that this RGW issue was fixed several releases ago. The OP’s cluster IIRC is mostly CephFS, so I suspect something else is going on.
On Oct 2, 2025, at 7:29 PM, Manuel Rios - EDH < mriosfer@easydatahost.com> wrote:
Hi,
Here user that suffer years ago a problem with orphans.
Years ago, after much research, we discovered that for some reason the WALLDB/Metadata entries were being deleted and corrupted, but the data on the disks weren't physically erased. Sometimes the garbage collector (deferred delete) would fail and skip the deletion, leaving hundreds of TB behind. Speaking with other heavy CEPH users, they were aware of this and couldn't find a great solution either, just they instead use replica 3 , used replica 4. (big customers with big budget) At the time, we were presented with two options: wipe each disk, and CEPH would only rebuild the data it knows is valid, but that would take time, maybe in your case where your full NVME will take not too much. Or create a new cluster and move the valid data.
In our case ceph orphan tool start looping due bugs and didn’t provide a real solution, our case 1PB ceph, with aprox 300TB orphaned.
I remember ceph orphan tool running for weeks ☹ bad ass time.
Our ceph use case : S3 and version 12 to 14 nautilus...
Sometimes , we as administrators doesn’t care about this issues until you need to wipe a lot of data. And you use simple calc and don’t match.
Regards,
-----Mensaje original----- De: Alexander Patrakov <patrakov@gmail.com> Enviado el: jueves, 2 de octubre de 2025 22:56 Para: Anthony D'Atri <anthony.datri@gmail.com> CC: ceph-users@ceph.io Asunto: [ceph-users] Re: Orphaned CephFS objects
On Thu, Oct 2, 2025 at 9:45 PM Anthony D'Atri < anthony.datri@gmail.com> wrote:
There is design work for a future ability to migrate a pool transparently, for example to effect a new EC profile, but that won't be available anytime soon.
This is, unfortunately, irrelevant in this case. Migrating a pool will migrate all the objects and their snapshots, even the unwanted ones. What Trey has (as far as I understood) is that there are some RADOS-level snapshots that do not correspond to any CephFS-level snapshots and are thus garbage, not to be migrated.
That's why the talk about file migration and not pool-level operations.
Now to the original question:
will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
Just tested in a test cluster - it won't balk and won't demand force even if you remove a pool that is actually used by files. So beware.
$ ceph osd pool create badfs_evilpool 32 ssd-only pool 'badfs_evilpool' created $ ceph fs add_data_pool badfs badfs_evilpool added data pool 38 to fsmap $ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_wrongpool cephfs_data_rightpool cephfs_data_hdd ] name: badfs, metadata pool: badfs_metadata, data pools: [badfs_data badfs_evilpool ] $ cephfs-shell -f badfs CephFS:~/>>> ls dir1/ dir2/ CephFS:~/>>> mkdir evil CephFS:~/>>> setxattr evil ceph.dir.layout.pool badfs_evilpool ceph.dir.layout.pool is successfully set to badfs_evilpool CephFS:~/>>> put /usr/bin/ls /evil/ls $ ceph fs rm_data_pool badfs badfs_evilpool removed data pool 38 from fsmap
-- Alexander Patrakov _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Fri, Oct 3, 2025 at 11:21 AM Dan van der Ster <dan.vanderster@clyso.com> wrote:
Hi Trey,
Greg's probably correct that a snap trim bug made these snapshots
untrimable.
(possibly you set osd_pg_max_concurrent_snap_trims = 0 ? see https://tracker.ceph.com/issues/54396)
There is one other very relevant MDS issue in this area that's been on my todo for awhile -- so I'll do a mini-brain dump here for the record. (Unless someone else already fixed this).
tl;dr MDS purges large files very very slowly, and there's a trivial fix:
ceph config set mds filer_max_purge_ops 40
The MDS has a pretty tight throttle on how quickly it purges rados after files have been deleted by a client. This is particularly noticeable if users delete very large files.
You can see the state of the "purge queue" by looking at perf dump output of the relevant active MDS. For example, from a previous case like this I worked on:
{ "pq_executing_ops": 480372, "pq_executing_ops_high_water": 480436, "pq_executing": 1, "pq_executing_high_water": 56, "pq_executed": 30003, "pq_item_in_journal": 66734 }
In that example -- the purge queue contains 66734 items (i.e. files) that the MDS needs to delete. Each file is striped across several 4MB objects, and files are purged one at a time. (That's the meaning of pq_executing = 1 there) In this case, the number of rados objects to delete, for that 1 file, is pq_executing_ops =480372. So that file has 480372 x 4MB parts == 1.92TB.
The MDS purges those underlying rados objects by sending `filer_max_purge_ops` in parallel to the OSDs. The default filer_max_purge_ops is only 10 -- meaning that the MDS is at most only ever asking 10 OSDs to delete rados objects. I've found that increasing filer_max_purge_ops to 40 is a good fix, even for very large active clusters.
Greg: Maybe we should change the default to 40 -- or even better, the MDS should use something like pg_num/2 or pg_num/4 for the data pool, so that effective filer purge ops will auto-scale with cluster size.
I think some kind of change here would be very sensible, but I'm not sure how to formulate it. Do we just want to deprecate filer_max_purg_ops and introduce a new config filer_pgs_per_purge_op or something? We can't increase a static default because the size of a Ceph cluster varies so much — many of the Kubernetes deployments are only 3 OSDs. :/ On Mon, Oct 6, 2025 at 9:02 AM Trey Palmer <nerdmagicatl@gmail.com> wrote:
Frédéric,
Thanks so much for looking into this.
The documentation isn't all that clear, but my impression has been that pool snapshots are an entirely different thing from CephFS snapshots.
At least the documentation says this, and it sounds from the bug report you posted like it's dealing with mon-managed snapshots?
To avoid snap id collision between mon-managed snapshots and file system snapshots, pools with mon-managed snapshots are not allowed to be attached to a file system. Also, mon-managed snapshots can’t be created in pools already attached to a file system either.
I'd love for my impression to be incorrect and to be able to fix it this way, though!
Thanks again,
Trey
On Mon, Oct 6, 2025 at 10:26 AM Frédéric Nass <frederic.nass@clyso.com> wrote:
Hi Greg,
This one? https://tracker.ceph.com/issues/64646
Symptoms: - CLONES are reported on 'rados df' while pool has no snapshots. - 'rados lssnap -p <pool_name>' command shows no snapshots but some clones are listed by 'rados listsnaps -p <pool_name> <object_name>' even sometimes with no 'head' object.
@Trey, if this is the one --- make sure it is before running the command --- running a 'ceph osd pool force-remove-snap <pool_name>' should put all leaked clone objects back in the trim queue and the OSDs should get rid of them.
Right, you probably aren't using pool snapshots. There were a couple of issues like this and Matan might know the others, though? -Greg
Just tested in a test cluster - it won't balk and won't demand force even if you remove a pool that is actually used by files. So beware.
Thanks very much, this is exactly what I needed to know! My next question is, what happens MDS-wise if you *do* remove a pool that still contains some file objects? This pool contains over 4 PB net, and it doesn't seem trivial to make sure I've transferred every single file object out of it, although it's not so hard to make sure all the *important* stuff has been moved. Ideally I'd just move everything to new CephFS instances and delete the original one along with its pools, but there's one set of data using the greater part of a PB that it's difficult to do that with and our current plan is to leave it on the original CephFS instance. This is made easy because that data rolls over in less than a quarter, so I just created a new pool and set dir fattr, and that should take care of getting that data out of the pool in pretty short order (obviously transitioning this quantity of data to new pools, while live in production, is not something that can be done in a matter of a few days or even a few weeks). Thanks again for the test! Trey Palmer On Thu, Oct 2, 2025 at 4:56 PM Alexander Patrakov <patrakov@gmail.com> wrote:
On Thu, Oct 2, 2025 at 9:45 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
There is design work for a future ability to migrate a pool transparently, for example to effect a new EC profile, but that won't be available anytime soon.
This is, unfortunately, irrelevant in this case. Migrating a pool will migrate all the objects and their snapshots, even the unwanted ones. What Trey has (as far as I understood) is that there are some RADOS-level snapshots that do not correspond to any CephFS-level snapshots and are thus garbage, not to be migrated.
That's why the talk about file migration and not pool-level operations.
Now to the original question:
will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
Just tested in a test cluster - it won't balk and won't demand force even if you remove a pool that is actually used by files. So beware.
$ ceph osd pool create badfs_evilpool 32 ssd-only pool 'badfs_evilpool' created $ ceph fs add_data_pool badfs badfs_evilpool added data pool 38 to fsmap $ ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_wrongpool cephfs_data_rightpool cephfs_data_hdd ] name: badfs, metadata pool: badfs_metadata, data pools: [badfs_data badfs_evilpool ] $ cephfs-shell -f badfs CephFS:~/>>> ls dir1/ dir2/ CephFS:~/>>> mkdir evil CephFS:~/>>> setxattr evil ceph.dir.layout.pool badfs_evilpool ceph.dir.layout.pool is successfully set to badfs_evilpool CephFS:~/>>> put /usr/bin/ls /evil/ls $ ceph fs rm_data_pool badfs badfs_evilpool removed data pool 38 from fsmap
-- Alexander Patrakov
Hi Trey, Greg's probably correct that a snap trim bug made these snapshots untrimable. (possibly you set osd_pg_max_concurrent_snap_trims = 0 ? see https://tracker.ceph.com/issues/54396) There is one other very relevant MDS issue in this area that's been on my todo for awhile -- so I'll do a mini-brain dump here for the record. (Unless someone else already fixed this). tl;dr MDS purges large files very very slowly, and there's a trivial fix: ceph config set mds filer_max_purge_ops 40 The MDS has a pretty tight throttle on how quickly it purges rados after files have been deleted by a client. This is particularly noticeable if users delete very large files. You can see the state of the "purge queue" by looking at perf dump output of the relevant active MDS. For example, from a previous case like this I worked on: { "pq_executing_ops": 480372, "pq_executing_ops_high_water": 480436, "pq_executing": 1, "pq_executing_high_water": 56, "pq_executed": 30003, "pq_item_in_journal": 66734 } In that example -- the purge queue contains 66734 items (i.e. files) that the MDS needs to delete. Each file is striped across several 4MB objects, and files are purged one at a time. (That's the meaning of pq_executing = 1 there) In this case, the number of rados objects to delete, for that 1 file, is pq_executing_ops =480372. So that file has 480372 x 4MB parts == 1.92TB. The MDS purges those underlying rados objects by sending `filer_max_purge_ops` in parallel to the OSDs. The default filer_max_purge_ops is only 10 -- meaning that the MDS is at most only ever asking 10 OSDs to delete rados objects. I've found that increasing filer_max_purge_ops to 40 is a good fix, even for very large active clusters. Greg: Maybe we should change the default to 40 -- or even better, the MDS should use something like pg_num/2 or pg_num/4 for the data pool, so that effective filer purge ops will auto-scale with cluster size. Hope that helps, Dan -- Dan van der Ster Ceph Executive Council | CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com On Thu, Oct 2, 2025 at 6:32 AM Trey Palmer <nerdmagicatl@gmail.com> wrote:
Hi,
Some months ago we deleted about 1.4PB net of CephFS data. Approximately 600TB net of the space we expected to reclaim, did not get reclaimed.
The cluster in question is now on Reef, but it was on Pacific when this happened. The cluster has 660 15TB NVMe OSD's on 55 nodes, plus 5 separate mon nodes. We almost exclusively use it for CephFS, though there is a small RBD pool for VM images. At the time we had a single MDS for the entire cluster, but we are now breaking that CephFS instance apart into multiple instances to spread the load (we tried multiple MDS ranks but it regressed performance and caused a near-disaster).
Anyway, we often delete ~500TB at a time and we haven't previously run into this orphaned object problem. At least, not at a scale big enough for us to notice -- generally I've gotten back the space I expected to free up.
Also we have a 30PB raw spinning disk cluster running Quincy that we deleted 2 PB net from at the same time, and it didn't exhibit this problem. So my assumption is that there is a bug in the version we were running that we exposed by deleting so much data at once.
Our support vendor did some digging using some special utilities and found that there are a bunch of orphaned objects from 2 particular snapshots, where the MDS has no reference to the objects, but they still exist on disk.
They feel that it would be somewhat difficult and risky to try to find and delete these objects via rados. However, obviously, 600+ *net* TB of all-NVMe storage is quite a lot of money to just let go to waste. That's effectively over 1.1 PB, once you figure EC overhead and the need to not fill the cluster over about 70%.
So to reclaim the pool I have created a plan to move everything off of the pool, and then delete the pool. Which we are largely needing to do anyway, due to splitting to multiple CephFS instances.
My question is, am I likely to run into problems with this? For example, will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
Thanks a lot for any insight,
Trey Palmer _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Dan. osd_pg_max_concurrent_snap_trims is set to the default of 1. filer_max_purge_ops is 300. We pretty routinely delete around 500TB of data before later deleting the snapshots of it, and that hasn't caused this problem before. On Fri, Oct 3, 2025 at 2:21 PM Dan van der Ster <dan.vanderster@clyso.com> wrote:
Hi Trey,
Greg's probably correct that a snap trim bug made these snapshots untrimable. (possibly you set osd_pg_max_concurrent_snap_trims = 0 ? see https://tracker.ceph.com/issues/54396)
There is one other very relevant MDS issue in this area that's been on my todo for awhile -- so I'll do a mini-brain dump here for the record. (Unless someone else already fixed this).
tl;dr MDS purges large files very very slowly, and there's a trivial fix:
ceph config set mds filer_max_purge_ops 40
The MDS has a pretty tight throttle on how quickly it purges rados after files have been deleted by a client. This is particularly noticeable if users delete very large files.
You can see the state of the "purge queue" by looking at perf dump output of the relevant active MDS. For example, from a previous case like this I worked on:
{ "pq_executing_ops": 480372, "pq_executing_ops_high_water": 480436, "pq_executing": 1, "pq_executing_high_water": 56, "pq_executed": 30003, "pq_item_in_journal": 66734 }
In that example -- the purge queue contains 66734 items (i.e. files) that the MDS needs to delete. Each file is striped across several 4MB objects, and files are purged one at a time. (That's the meaning of pq_executing = 1 there) In this case, the number of rados objects to delete, for that 1 file, is pq_executing_ops =480372. So that file has 480372 x 4MB parts == 1.92TB.
The MDS purges those underlying rados objects by sending `filer_max_purge_ops` in parallel to the OSDs. The default filer_max_purge_ops is only 10 -- meaning that the MDS is at most only ever asking 10 OSDs to delete rados objects. I've found that increasing filer_max_purge_ops to 40 is a good fix, even for very large active clusters.
Greg: Maybe we should change the default to 40 -- or even better, the MDS should use something like pg_num/2 or pg_num/4 for the data pool, so that effective filer purge ops will auto-scale with cluster size.
Hope that helps,
Dan
-- Dan van der Ster Ceph Executive Council | CTO @ CLYSO Try our Ceph Analyzer -- https://analyzer.clyso.com/ https://clyso.com | dan.vanderster@clyso.com
On Thu, Oct 2, 2025 at 6:32 AM Trey Palmer <nerdmagicatl@gmail.com> wrote:
Hi,
Some months ago we deleted about 1.4PB net of CephFS data. Approximately 600TB net of the space we expected to reclaim, did not get reclaimed.
The cluster in question is now on Reef, but it was on Pacific when this happened. The cluster has 660 15TB NVMe OSD's on 55 nodes, plus 5
mon nodes. We almost exclusively use it for CephFS, though there is a small RBD pool for VM images. At the time we had a single MDS for the entire cluster, but we are now breaking that CephFS instance apart into multiple instances to spread the load (we tried multiple MDS ranks but it regressed performance and caused a near-disaster).
Anyway, we often delete ~500TB at a time and we haven't previously run into this orphaned object problem. At least, not at a scale big enough for us to notice -- generally I've gotten back the space I expected to free up.
Also we have a 30PB raw spinning disk cluster running Quincy that we deleted 2 PB net from at the same time, and it didn't exhibit this problem. So my assumption is that there is a bug in the version we were running that we exposed by deleting so much data at once.
Our support vendor did some digging using some special utilities and found that there are a bunch of orphaned objects from 2 particular snapshots, where the MDS has no reference to the objects, but they still exist on disk.
They feel that it would be somewhat difficult and risky to try to find and delete these objects via rados. However, obviously, 600+ *net* TB of all-NVMe storage is quite a lot of money to just let go to waste. That's effectively over 1.1 PB, once you figure EC overhead and the need to not fill the cluster over about 70%.
So to reclaim the pool I have created a plan to move everything off of
separate the
pool, and then delete the pool. Which we are largely needing to do anyway, due to splitting to multiple CephFS instances.
My question is, am I likely to run into problems with this? For example, will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk?
Thanks a lot for any insight,
Trey Palmer _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Alexander Patrakov
-
Anthony D'Atri
-
Dan van der Ster
-
Frédéric Nass
-
Gregory Farnum
-
Manuel Rios - EDH
-
Trey Palmer