Request for Recommendations: Tiering/Archiving from NetApp to Ceph (with stub file support)
Hi all, We're exploring solutions to offload large volumes of data (on the order of petabytes) from our NetApp all-flash storage to our more cost-effective, HDD-based Ceph storage cluster, based on criteria such as: last access time older than X years. Ideally, we would like to leave behind a 'stub' or placeholder file on the NetApp side to preserve the original directory structure and potentially enable some sort of transparent access or recall if needed. This kind of setup is commonly supported by solutions like DataCore/FileFly, but as far as we can tell, FileFly doesn’t support Ceph as a backend and instead favors its own Swarm object store. Has anyone here implemented a similar tiering/archive/migration solution involving NetApp and Ceph? We’re specifically looking for: * Enterprise-grade tooling * Stub file support or similar metadata-preserving offload * Support and reliability (given the scale, we can’t afford data loss or inconsistency) * Either commercial or well-supported open source solutions Any do’s/don’ts, war stories, or product recommendations would be greatly appreciated. We’re open to paying for software or services if it brings us the reliability and integration we need. Thanks in advance! MJ
Not a full answer to what you’re looking for, but something adjacent: https://ceph.io/en/news/blog/2025/rgw-tiering-enhancements-part1/ https://ceph.io/en/news/blog/2025/rgw-tiering-enhancements-part2/ <https://ceph.io/en/news/blog/2025/rgw-tiering-enhancements-part2/> I’m not sure if/when this will appear in a Squid release — maybe it’s already there — but I have to imagine it will be in Tentacle. Perhaps you could invert the usual tiering logic and have your clients point at the Ceph RGW endpoint, but I suspect that wouldn’t meet you performance needs. However, here https://www.netapp.com/data-services/tiering/ I see a NetApp feature called BlueXP that appears to do this sort of tiering. The description is a big vague, but it speaks of cloud object storage and mentions AWS, so it seems quite plausible that it could be pointed at an RGW endpoint for tiering.
Hi all,
We're exploring solutions to offload large volumes of data (on the order of petabytes) from our NetApp all-flash storage to our more cost-effective, HDD-based Ceph storage cluster, based on criteria such as: last access time older than X years.
Ideally, we would like to leave behind a 'stub' or placeholder file on the NetApp side to preserve the original directory structure and potentially enable some sort of transparent access or recall if needed. This kind of setup is commonly supported by solutions like DataCore/FileFly, but as far as we can tell, FileFly doesn’t support Ceph as a backend and instead favors its own Swarm object store.
Has anyone here implemented a similar tiering/archive/migration solution involving NetApp and Ceph?
We’re specifically looking for:
* Enterprise-grade tooling
* Stub file support or similar metadata-preserving offload
* Support and reliability (given the scale, we can’t afford data loss or inconsistency)
* Either commercial or well-supported open source solutions
Any do’s/don’ts, war stories, or product recommendations would be greatly appreciated. We’re open to paying for software or services if it brings us the reliability and integration we need.
Thanks in advance!
MJ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi MJ, Not sure if it's the right direction but backup software (like for example Commvault) has an archiving option with stub support. You can configure the s3 cloud library with CEPH s3 as a backend. I've never tested it - just a thought. On Mon, 5 May 2025 at 00:49, sacawulu <cyusedfzfb@gmail.com> wrote:
Hi all,
We're exploring solutions to offload large volumes of data (on the order of petabytes) from our NetApp all-flash storage to our more cost-effective, HDD-based Ceph storage cluster, based on criteria such as: last access time older than X years.
Ideally, we would like to leave behind a 'stub' or placeholder file on the NetApp side to preserve the original directory structure and potentially enable some sort of transparent access or recall if needed. This kind of setup is commonly supported by solutions like DataCore/FileFly, but as far as we can tell, FileFly doesn’t support Ceph as a backend and instead favors its own Swarm object store.
Has anyone here implemented a similar tiering/archive/migration solution involving NetApp and Ceph?
We’re specifically looking for:
* Enterprise-grade tooling
* Stub file support or similar metadata-preserving offload
* Support and reliability (given the scale, we can’t afford data loss or inconsistency)
* Either commercial or well-supported open source solutions
Any do’s/don’ts, war stories, or product recommendations would be greatly appreciated. We’re open to paying for software or services if it brings us the reliability and integration we need.
Thanks in advance!
MJ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Łukasz Borek lukasz@borek.org.pl
There are likely simpler answers if you want to tier entire buckets, but it sounds like you are hosting a filesystem(s) on NetApp and want to tier them. It would be nice to have NetApp running Ceph as a block store, but I don't think crush is sophisticated enough to migrate components of a filesystem pool based on the ages of files/directories in them. For one thing, I'm not sure that the PGs in the pool can/should be that aware of such details or that you might not get into problems with fragments of files in different PGs with much of a PG being un-aged data. So I'm not optimistic on that concept. What that suggests to me is that you might use an overlay filesystem, where the different tiers overlay each other to present a unified filesystem image. This is precisely what containers do, although much of their goal is simply optimising shared image layers. A variation of this is Copy-on-Write (COW), but what you want is more like the reverse. At any rate, a frontend overlay filesystem with NetApp overlaying a secondary Ceph system seems like a likely solution. Then all you'd need would be a mechanism to move aged-out resources. That might even be a good use of rsync. Tim On 5/4/25 10:20, sacawulu wrote:
Hi all,
We're exploring solutions to offload large volumes of data (on the order of petabytes) from our NetApp all-flash storage to our more cost-effective, HDD-based Ceph storage cluster, based on criteria such as: last access time older than X years.
Ideally, we would like to leave behind a 'stub' or placeholder file on the NetApp side to preserve the original directory structure and potentially enable some sort of transparent access or recall if needed. This kind of setup is commonly supported by solutions like DataCore/FileFly, but as far as we can tell, FileFly doesn’t support Ceph as a backend and instead favors its own Swarm object store.
Has anyone here implemented a similar tiering/archive/migration solution involving NetApp and Ceph?
We’re specifically looking for:
* Enterprise-grade tooling
* Stub file support or similar metadata-preserving offload
* Support and reliability (given the scale, we can’t afford data loss or inconsistency)
* Either commercial or well-supported open source solutions
Any do’s/don’ts, war stories, or product recommendations would be greatly appreciated. We’re open to paying for software or services if it brings us the reliability and integration we need.
Thanks in advance!
MJ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi everyone, Many thanks for the thoughtful and insightful replies to our earlier question—both on- and off-list. We really appreciate the input; it's been very valuable. For the benefit of others and for the archives, I wanted to mention two products that came up and seem promising: FileFly (we still have to confirm Ceph S3 compatinility) Komprise Both are very much enterprise-grade solutions and appear capable of meeting our needs. On a related note, we’re also exploring immutable Veeam backups to Ceph S3-compatible object storage. We came across this unofficial Veeam object storage compatibility list: https://forums.veeam.com/object-storage-as-backup-target-f52/unoffizial-comp... It looks promising. We've also read that Veeam/ceph works better when veeam creates larger objects rather than many small ones—something we’ll aim to validate during our PoC. If anyone here has experience using Veeam with S3-compatible object storage/immutable backups, we’d love to hear about it. If this drifts too far from the main topic of the list, feel free to reply off-list. Thanks again, have a nice day everybody! Best regards, MJ On 5/6/25 00:42, Tim Holloway wrote:
There are likely simpler answers if you want to tier entire buckets, but it sounds like you are hosting a filesystem(s) on NetApp and want to tier them. It would be nice to have NetApp running Ceph as a block store, but I don't think crush is sophisticated enough to migrate components of a filesystem pool based on the ages of files/directories in them. For one thing, I'm not sure that the PGs in the pool can/should be that aware of such details or that you might not get into problems with fragments of files in different PGs with much of a PG being un-aged data. So I'm not optimistic on that concept.
What that suggests to me is that you might use an overlay filesystem, where the different tiers overlay each other to present a unified filesystem image. This is precisely what containers do, although much of their goal is simply optimising shared image layers. A variation of this is Copy-on-Write (COW), but what you want is more like the reverse.
At any rate, a frontend overlay filesystem with NetApp overlaying a secondary Ceph system seems like a likely solution. Then all you'd need would be a mechanism to move aged-out resources. That might even be a good use of rsync.
Tim
On 5/4/25 10:20, sacawulu wrote:
Hi all,
We're exploring solutions to offload large volumes of data (on the order of petabytes) from our NetApp all-flash storage to our more cost-effective, HDD-based Ceph storage cluster, based on criteria such as: last access time older than X years.
Ideally, we would like to leave behind a 'stub' or placeholder file on the NetApp side to preserve the original directory structure and potentially enable some sort of transparent access or recall if needed. This kind of setup is commonly supported by solutions like DataCore/FileFly, but as far as we can tell, FileFly doesn’t support Ceph as a backend and instead favors its own Swarm object store.
Has anyone here implemented a similar tiering/archive/migration solution involving NetApp and Ceph?
We’re specifically looking for:
* Enterprise-grade tooling
* Stub file support or similar metadata-preserving offload
* Support and reliability (given the scale, we can’t afford data loss or inconsistency)
* Either commercial or well-supported open source solutions
Any do’s/don’ts, war stories, or product recommendations would be greatly appreciated. We’re open to paying for software or services if it brings us the reliability and integration we need.
Thanks in advance!
MJ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On a related note, we’re also exploring immutable Veeam backups to Ceph S3-compatible object storage. We came across this unofficial Veeam object storage compatibility list: https://forums.veeam.com/object-storage-as-backup-target-f52/unoffizial-comp...
It looks promising. We've also read that Veeam/ceph works better when veeam creates larger objects rather than many small ones—something we’ll aim to validate during our PoC.
If anyone here has experience using Veeam with S3-compatible object storage/immutable backups, we’d love to hear about it. If this drifts too far from the main topic of the list, feel free to reply off-list.
It works fine. As you said, make sure to set Veeam to use the largest possible object size, otherwise ceph will be spending a lot of time shuffling small items around, and the overhead will be what kills you, not the actual size. First attempt for us ended in ceph having some 660M objects of average size less than 180k. Probably a bit outdated but here is a guide on where to set the sizes: https://community.veeam.com/blogs-and-podcasts-57/sobr-veeam-capacity-tier-c... -- May the most significant bit of your life be positive.
participants (5)
-
Anthony D'Atri
-
Janne Johansson
-
Lukasz Borek
-
sacawulu
-
Tim Holloway