5 Feb
2020
5 Feb
'20
8:27 p.m.
HI all,
what is the best approach for OSD backups and recovery?
We use only Radosgw with S3 API and I need to backup the content of S3 buckets.
Currently I sync s3 buckets to local filesystem and backup the content using Amanda.
I believe that there must a better way to do this but I couldn't find it in docs.
I know that one option is to setup an archive zone, but it requires an additional ceph
cluster that needs to be maintained and looked after. I would rather avoid that.
How can I backup an entire Ceph cluster? Or individual OSDs in the way that will allow me
to recover the data correctly?
Many thanks,Ludek
30 May
30 May
3:01 a.m.
Ludek;
As a cluster system, Ceph isn't really intended to be backed up. It's designed to
take quite a beating, and preserve your data.
From a broader disaster recovery perspective, here's how I architected my clusters:
Our primary cluster is laid out in such a way that an entire rack can fail without read /
write being impacted, much less data integrity. On top of that, our RadosGW was a
multi-site setup which automatically sends a copy of every object to a second cluster at a
different location.
Thus my disaster recovery looks like this:
1 rack or less: no user impact, rebuild rack
2 racks: users are unable to add objects, but existing data is safe, rebuild cluster (or
as below)
Whole site: switch second site to master and continue
No backup or recovery necessary.
You might look the multi-site documentation:
https://docs.ceph.com/docs/master/radosgw/multisite/
I had a long conversation with our owner on this same topic, and how the organization
would have to move from a "Backup & Recover" mindset to a "Disaster
Recovery" mindset. It worked well for us, as we were looking to move more towards
Risk Analysis based approaches anyway.
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International, Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: Ludek Navratil [mailto:ludek.navratil@yahoo.co.uk]
Sent: Wednesday, February 5, 2020 6:57 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] OSD backups and recovery
HI all,
what is the best approach for OSD backups and recovery? We use only Radosgw with S3 API
and I need to backup the content of S3 buckets. Currently I sync s3 buckets to local
filesystem and backup the content using Amanda.
I believe that there must a better way to do this but I couldn't find it in docs.
I know that one option is to setup an archive zone, but it requires an additional ceph
cluster that needs to be maintained and looked after. I would rather avoid that.
How can I backup an entire Ceph cluster? Or individual OSDs in the way that will allow me
to recover the data correctly?
Many thanks,Ludek
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
3:15 a.m.
Am Fr., 29. Mai 2020 um 23:32 Uhr schrieb <DHilsbos(a)performair.com>om>:
Ludek;
As a cluster system, Ceph isn't really intended to be backed up. It's
designed to take quite a beating, and preserve your data.
But that does not save me when a crypto trojan encrypts all my data. There
should always be an offline backup that can be restored in case of crypto
trojan
From a broader disaster recovery perspective,
here's how I architected my
clusters:
Our primary cluster is laid out in such a way that an entire rack can fail
without read / write being impacted, much less data integrity. On top of
that, our RadosGW was a multi-site setup which automatically sends a copy
of every object to a second cluster at a different location.
Thus my disaster recovery looks like this:
1 rack or less: no user impact, rebuild rack
2 racks: users are unable to add objects, but existing data is safe,
rebuild cluster (or as below)
Whole site: switch second site to master and continue
No backup or recovery necessary.
You might look the multi-site documentation:
https://docs.ceph.com/docs/master/radosgw/multisite/
I had a long conversation with our owner on this same topic, and how the
organization would have to move from a "Backup & Recover" mindset to a
"Disaster Recovery" mindset. It worked well for us, as we were looking to
move more towards Risk Analysis based approaches anyway.
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International, Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: Ludek Navratil [mailto:ludek.navratil@yahoo.co.uk]
Sent: Wednesday, February 5, 2020 6:57 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] OSD backups and recovery
HI all,
what is the best approach for OSD backups and recovery? We use only
Radosgw with S3 API and I need to backup the content of S3 buckets.
Currently I sync s3 buckets to local filesystem and backup the content
using Amanda.
I believe that there must a better way to do this but I couldn't find it
in docs.
I know that one option is to setup an archive zone, but it requires an
additional ceph cluster that needs to be maintained and looked after. I
would rather avoid that.
How can I backup an entire Ceph cluster? Or individual OSDs in the way
that will allow me to recover the data correctly?
Many thanks,Ludek
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:31 a.m.
SpiderFox;
If you're concerned about ransomware (and you should be), then you should:
a) protect the cluster from the internet AND from USERS.
b) place another technology between your cluster and your users (I use Nextcloud backed by
RadosGW through S3 buckets)
c) turn on versioning in your buckets
If you have a backup solution that can handle petabytes of data reliably, then certainly
use it. Everything I've tried fell over dead at a couple dozen terabytes.
Nothing is fool proof, not even the vaunted offline backup (ever try to do a recover and
find the tape can't be read?).
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
DHilsbos(a)PerformAir.com
300 S. Hamilton Pl.
Gilbert, AZ 85233
Phone: (480) 610-3500
Fax: (480) 610-3501
www.PerformAir.com
-----Original Message-----
From: Coding SpiderFox [mailto:codingspiderfox@gmail.com]
Sent: Friday, May 29, 2020 2:45 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: OSD backups and recovery
Am Fr., 29. Mai 2020 um 23:32 Uhr schrieb <DHilsbos(a)performair.com>om>:
Ludek;
As a cluster system, Ceph isn't really intended to be backed up. It's
designed to take quite a beating, and preserve your data.
But that does not save me when a crypto trojan encrypts all my data. There should
always be an offline backup that can be restored in case of crypto trojan
From a broader disaster recovery perspective,
here's how I architected
my
clusters:
Our primary cluster is laid out in such a way that an entire rack can
fail without read / write being impacted, much less data integrity.
On top of that, our RadosGW was a multi-site setup which automatically
sends a copy of every object to a second cluster at a different location.
Thus my disaster recovery looks like this:
1 rack or less: no user impact, rebuild rack
2 racks: users are unable to add objects, but existing data is safe,
rebuild cluster (or as below) Whole site: switch second site to master
and continue
No backup or recovery necessary.
You might look the multi-site documentation:
https://docs.ceph.com/docs/master/radosgw/multisite/
I had a long conversation with our owner on this same topic, and how
the organization would have to move from a "Backup & Recover" mindset
to a "Disaster Recovery" mindset. It worked well for us, as we were
looking to move more towards Risk Analysis based approaches anyway.
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International, Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: Ludek Navratil [mailto:ludek.navratil@yahoo.co.uk]
Sent: Wednesday, February 5, 2020 6:57 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] OSD backups and recovery
HI all,
what is the best approach for OSD backups and recovery? We use only
Radosgw with S3 API and I need to backup the content of S3 buckets.
Currently I sync s3 buckets to local filesystem and backup the content
using Amanda.
I believe that there must a better way to do this but I couldn't find
it in docs.
I know that one option is to setup an archive zone, but it requires an
additional ceph cluster that needs to be maintained and looked after.
I would rather avoid that.
How can I backup an entire Ceph cluster? Or individual OSDs in the way
that will allow me to recover the data correctly?
Many thanks,Ludek
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
5:31 a.m.
For some reason I’d thought replication between clusters was an “official” method of
backing up.
On May 29, 2020, at 4:31 PM,
<DHilsbos(a)performair.com> <DHilsbos(a)performair.com> wrote:
Ludek;
As a cluster system, Ceph isn't really intended to be backed up. It's designed
to take quite a beating, and preserve your data.
From a broader disaster recovery perspective, here's how I architected my clusters:
Our primary cluster is laid out in such a way that an entire rack can fail without read /
write being impacted, much less data integrity. On top of that, our RadosGW was a
multi-site setup which automatically sends a copy of every object to a second cluster at a
different location.
Thus my disaster recovery looks like this:
1 rack or less: no user impact, rebuild rack
2 racks: users are unable to add objects, but existing data is safe, rebuild cluster (or
as below)
Whole site: switch second site to master and continue
No backup or recovery necessary.
You might look the multi-site documentation:
https://docs.ceph.com/docs/master/radosgw/multisite/
I had a long conversation with our owner on this same topic, and how the organization
would have to move from a "Backup & Recover" mindset to a "Disaster
Recovery" mindset. It worked well for us, as we were looking to move more towards
Risk Analysis based approaches anyway.
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International, Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: Ludek Navratil [mailto:ludek.navratil@yahoo.co.uk]
Sent: Wednesday, February 5, 2020 6:57 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] OSD backups and recovery
HI all,
what is the best approach for OSD backups and recovery? We use only Radosgw with S3 API
and I need to backup the content of S3 buckets. Currently I sync s3 buckets to local
filesystem and backup the content using Amanda.
I believe that there must a better way to do this but I couldn't find it in docs.
I know that one option is to setup an archive zone, but it requires an additional ceph
cluster that needs to be maintained and looked after. I would rather avoid that.
How can I backup an entire Ceph cluster? Or individual OSDs in the way that will allow me
to recover the data correctly?
Many thanks,Ludek
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:56 a.m.
Jarett;
It is and it isn't. Replication can be thought of as continuous backups.
Backups, especially as SpiderFox is suggesting, are point-in-time, immutable copies of
data. Until they are written over, they don't change, even if the data does.
In Ceph's RadosGW (RGW) multi-site replication changes, even "bad" changes,
are pushed to the peer as quickly as the system can manage. Even the
"replication" occurring within a cluster can be considered a "backup,"
sort of.
As SpiderFox suggested, if malware is able to delete or encrypt the files, either through
RGW, RADOS, or on the underlying block device, you've got problems. Note though; if
they bypass RGW, then (AFAIK), the changes won't be replicated to the peer.
That's why I talk about disaster recovery. Backing up is one disaster recovery
technique, and is still perfectly valid. Perform Air maintains backups of our Active
Directory domain controllers, for instance.
Clustering, and off-site replication, are other disaster recovery paradigms. Each has
advantages and disadvantages.
Ultimately, as long as everything works, I believe the only wrong disaster recovery plan
is doing nothing.
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International, Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: Jarett DeAngelis [mailto:jarett@reticulum.us]
Sent: Friday, May 29, 2020 5:02 PM
To: Dominic Hilsbos
Cc: ludek.navratil(a)yahoo.co.uk; ceph-users(a)ceph.io
Subject: Re: [ceph-users] OSD backups and recovery
For some reason I’d thought replication between clusters was an “official” method of
backing up.
On May 29, 2020, at 4:31 PM,
<DHilsbos(a)performair.com> <DHilsbos(a)performair.com> wrote:
Ludek;
As a cluster system, Ceph isn't really intended to be backed up. It's designed
to take quite a beating, and preserve your data.
From a broader disaster recovery perspective, here's how I architected my clusters:
Our primary cluster is laid out in such a way that an entire rack can fail without read /
write being impacted, much less data integrity. On top of that, our RadosGW was a
multi-site setup which automatically sends a copy of every object to a second cluster at a
different location.
Thus my disaster recovery looks like this:
1 rack or less: no user impact, rebuild rack
2 racks: users are unable to add objects, but existing data is safe,
rebuild cluster (or as below) Whole site: switch second site to master
and continue
No backup or recovery necessary.
You might look the multi-site documentation:
https://docs.ceph.com/docs/master/radosgw/multisite/
I had a long conversation with our owner on this same topic, and how the organization
would have to move from a "Backup & Recover" mindset to a "Disaster
Recovery" mindset. It worked well for us, as we were looking to move more towards
Risk Analysis based approaches anyway.
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International, Inc.
DHilsbos(a)PerformAir.com
www.PerformAir.com
-----Original Message-----
From: Ludek Navratil [mailto:ludek.navratil@yahoo.co.uk]
Sent: Wednesday, February 5, 2020 6:57 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] OSD backups and recovery
HI all,
what is the best approach for OSD backups and recovery? We use only Radosgw with S3 API
and I need to backup the content of S3 buckets. Currently I sync s3 buckets to local
filesystem and backup the content using Amanda.
I believe that there must a better way to do this but I couldn't find it in docs.
I know that one option is to setup an archive zone, but it requires an additional ceph
cluster that needs to be maintained and looked after. I would rather avoid that.
How can I backup an entire Ceph cluster? Or individual OSDs in the way that will allow me
to recover the data correctly?
Many thanks,Ludek
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1427
days inactive
1542
days old
5 comments
4 participants
participants (4)
-
Coding SpiderFox -
DHilsbos＠performair.com -
Jarett DeAngelis -
Ludek Navratil