20 Sep
2019
20 Sep
'19
7:59 p.m.
The question was posed, "What if we want to backup our RGW data to
tape?" Anyone doing this? Any suggestions? We could probably just
catch any PUT requests and queue them to be written to tape. Our
dataset is so large, that traditional backup solutions don't seem
feasible (GFS), so probably a single copy (or two copies on different
tapes at the same time) when the object is created.
Bonus points for being near-line.
Thanks,
Robert LeBlanc
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
20 Sep
20 Sep
8:10 p.m.
Probably easiest if you get a tape library that supports S3. You might
even have some luck with radosgw's cloud sync module (but I wouldn't
count on it, Octopus should improve things, though)
Just intercepting PUT requests isn't that easy because of multi-part
stuff and load balancing. I.e., if you upload a large file you should
be sending it in chunks and each chunk should go to a different
server, that makes any "simple" solutions pretty messy.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Fri, Sep 20, 2019 at 8:01 PM Robert LeBlanc <robert(a)leblancnet.us> wrote:
>
> The question was posed, "What if we want to backup our RGW data to
> tape?" Anyone doing this? Any suggestions? We could probably just
> catch any PUT requests and queue them to be written to tape. Our
> dataset is so large, that traditional backup solutions don't seem
> feasible (GFS), so probably a single copy (or two copies on different
> tapes at the same time) when the object is created.
>
> Bonus points for being near-line.
>
> Thanks,
> Robert LeBlanc
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:28 p.m.
On Fri, Sep 20, 2019 at 11:10 AM Paul Emmerich <paul.emmerich(a)croit.io> wrote:
Probably easiest if you get a tape library that supports S3. You might
even have some luck with radosgw's cloud sync module (but I wouldn't
count on it, Octopus should improve things, though)
Just intercepting PUT requests isn't that easy because of multi-part
stuff and load balancing. I.e., if you upload a large file you should
be sending it in chunks and each chunk should go to a different
server, that makes any "simple" solutions pretty messy.
I wasn't aware of any library being S3 aware, usually it's been part
of the backup software. Do you have any suggestions for multi PB
libraries that have the S3 feature?
The idea with the PUT was not to intercept them in the path, but to
basically have RGW log access to LogStash, then a job would run to
find all the objects that were PUT within a time frame, then read the
objects off the cluster and write them to tape. Maybe that's not as
easy as I'm thinking either.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
21 Sep
21 Sep
12:18 a.m.
Robert,
There're a storage company that integrate TAPES as OSD for deep-cold ceph.
But the code is not opensource
Regards
-----Mensaje original-----
De: Robert LeBlanc <robert(a)leblancnet.us>
Enviado el: viernes, 20 de septiembre de 2019 23:28
Para: Paul Emmerich <paul.emmerich(a)croit.io>
CC: ceph-users <ceph-users(a)ceph.io>
Asunto: [ceph-users] Re: RGW backup to tape
On Fri, Sep 20, 2019 at 11:10 AM Paul Emmerich <paul.emmerich(a)croit.io>
wrote:
Probably easiest if you get a tape library that supports S3. You might
even have some luck with radosgw's cloud sync module (but I wouldn't
count on it, Octopus should improve things, though)
Just intercepting PUT requests isn't that easy because of multi-part
stuff and load balancing. I.e., if you upload a large file you should
be sending it in chunks and each chunk should go to a different
server, that makes any "simple" solutions pretty messy.
I wasn't aware of any library being S3 aware, usually it's been part of the
backup software. Do you have any suggestions for multi PB libraries that
have the S3 feature?
The idea with the PUT was not to intercept them in the path, but to
basically have RGW log access to LogStash, then a job would run to find all
the objects that were PUT within a time frame, then read the objects off the
cluster and write them to tape. Maybe that's not as easy as I'm thinking
either.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email
to ceph-users-leave(a)ceph.io
1674
days inactive
1674
days old
3 comments
3 participants
participants (3)
-
EDH - Manuel Rios Fernandez -
Paul Emmerich -
Robert LeBlanc