The question was posed, "What if we want to backup our RGW data to tape?" Anyone doing this? Any suggestions? We could probably just catch any PUT requests and queue them to be written to tape. Our dataset is so large, that traditional backup solutions don't seem feasible (GFS), so probably a single copy (or two copies on different tapes at the same time) when the object is created. Bonus points for being near-line. Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
Probably easiest if you get a tape library that supports S3. You might even have some luck with radosgw's cloud sync module (but I wouldn't count on it, Octopus should improve things, though) Just intercepting PUT requests isn't that easy because of multi-part stuff and load balancing. I.e., if you upload a large file you should be sending it in chunks and each chunk should go to a different server, that makes any "simple" solutions pretty messy. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Fri, Sep 20, 2019 at 8:01 PM Robert LeBlanc <robert@leblancnet.us> wrote:
The question was posed, "What if we want to backup our RGW data to tape?" Anyone doing this? Any suggestions? We could probably just catch any PUT requests and queue them to be written to tape. Our dataset is so large, that traditional backup solutions don't seem feasible (GFS), so probably a single copy (or two copies on different tapes at the same time) when the object is created.
Bonus points for being near-line.
Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Fri, Sep 20, 2019 at 11:10 AM Paul Emmerich <paul.emmerich@croit.io> wrote:
Probably easiest if you get a tape library that supports S3. You might even have some luck with radosgw's cloud sync module (but I wouldn't count on it, Octopus should improve things, though)
Just intercepting PUT requests isn't that easy because of multi-part stuff and load balancing. I.e., if you upload a large file you should be sending it in chunks and each chunk should go to a different server, that makes any "simple" solutions pretty messy.
I wasn't aware of any library being S3 aware, usually it's been part of the backup software. Do you have any suggestions for multi PB libraries that have the S3 feature? The idea with the PUT was not to intercept them in the path, but to basically have RGW log access to LogStash, then a job would run to find all the objects that were PUT within a time frame, then read the objects off the cluster and write them to tape. Maybe that's not as easy as I'm thinking either. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
Robert, There're a storage company that integrate TAPES as OSD for deep-cold ceph. But the code is not opensource Regards -----Mensaje original----- De: Robert LeBlanc <robert@leblancnet.us> Enviado el: viernes, 20 de septiembre de 2019 23:28 Para: Paul Emmerich <paul.emmerich@croit.io> CC: ceph-users <ceph-users@ceph.io> Asunto: [ceph-users] Re: RGW backup to tape On Fri, Sep 20, 2019 at 11:10 AM Paul Emmerich <paul.emmerich@croit.io> wrote:
Probably easiest if you get a tape library that supports S3. You might even have some luck with radosgw's cloud sync module (but I wouldn't count on it, Octopus should improve things, though)
Just intercepting PUT requests isn't that easy because of multi-part stuff and load balancing. I.e., if you upload a large file you should be sending it in chunks and each chunk should go to a different server, that makes any "simple" solutions pretty messy.
I wasn't aware of any library being S3 aware, usually it's been part of the backup software. Do you have any suggestions for multi PB libraries that have the S3 feature? The idea with the PUT was not to intercept them in the path, but to basically have RGW log access to LogStash, then a job would run to find all the objects that were PUT within a time frame, then read the objects off the cluster and write them to tape. Maybe that's not as easy as I'm thinking either. ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
EDH - Manuel Rios Fernandez
-
Paul Emmerich
-
Robert LeBlanc