12 Dec
2023
12 Dec
'23
3:02 a.m.
Hi,
I'm developing RBD images' backup system. In my case, a backup data
must be stored at least two weeks. To meet this requirement, I'd like
to take backups as follows:
1. Take a full backup by rbd export first.
2. Take a differencial backups everyday.
3. Merge the full backup and the oldest (taken two weeks ago) diff.
As a result of evaluation, I confirmed there is no problem in step 1
and 2. However,
I found that step 3 couldn't be accomplished by `rbd merge-diff <full
backup> <diff>`
because `rbd merge-diff` only accepts a diff as a first parameter. Is
there any way
to merge a full backup and a diff?
Thanks,
Satoru
12 Dec
12 Dec
3:23 p.m.
On Tue, Dec 12, 2023 at 1:03 AM Satoru Takeuchi
<satoru.takeuchi(a)gmail.com> wrote:
Hi,
I'm developing RBD images' backup system. In my case, a backup data
must be stored at least two weeks. To meet this requirement, I'd like
to take backups as follows:
1. Take a full backup by rbd export first.
2. Take a differencial backups everyday.
3. Merge the full backup and the oldest (taken two weeks ago) diff.
As a result of evaluation, I confirmed there is no problem in step 1
and 2. However,
I found that step 3 couldn't be accomplished by `rbd merge-diff <full
backup> <diff>`
because `rbd merge-diff` only accepts a diff as a first parameter. Is
there any way
to merge a full backup and a diff?
Hi Satoru,
Not at the moment. Mykola has an old work-in-progress PR which extends
"rbd import-diff" command to make this possible [1]. Since you as
a user expected "rbd merge-diff" to be able to this, I wonder if this
functionality might be better placed under "rbd merge-diff"? That way
the operations on files would be separated from the operations on RBD
images.
For your backup system, couldn't you just merge the two oldest
differentials with "rbd merge-diff" instead? Instead of advancing the
full export, you would be advancing the first differential -- it would
represent a diff between the initial export and the "2 weeks" backup
over time.
[1] https://github.com/ceph/ceph/pull/41375
Thanks,
Ilya
13 Dec
13 Dec
2:48 a.m.
Hi Ilya,
2023年12月12日(火) 21:23 Ilya Dryomov <idryomov(a)gmail.com>om>:
Not at the moment. Mykola has an old work-in-progress
PR which extends
"rbd import-diff" command to make this possible [1].
I didn't know this PR. Thank you very much.I'll evaluate this PR later.
Since you as
a user expected "rbd merge-diff" to be able to this, I wonder if this
functionality might be better placed under "rbd merge-diff"? That way
the operations on files would be separated from the operations on RBD
images.
Yes, I think so too.
For your backup system, couldn't you just merge the two oldest
differentials with "rbd merge-diff" instead? Instead of advancing the
full export, you would be advancing the first differential -- it would
represent a diff between the initial export and the "2 weeks" backup
over time.
Yes, it's possible. It's one of a workaround I thought. Then the
backup data are as follows:
a. The full backup taken at least 14 days ago.
b. The latest 14 days backup data
In this case, I concern the effect if (a) becomes too old. However, it
might be a groundless fear.
I'll give it some more thought. Thank you again.
Best,
Satoru
7:06 p.m.
On Wed, Dec 13, 2023 at 12:48 AM Satoru Takeuchi
<satoru.takeuchi(a)gmail.com> wrote:
Hi Ilya,
2023年12月12日(火) 21:23 Ilya Dryomov <idryomov(a)gmail.com>om>:
I think it would be:
a. A full backup (taken potentially months ago, exact age doesn't
really matter)
b. Differential #1 - diff from the full backup to the 14 days old version
c. Differential #2 - diff from the 14 days old to the 13 days old version
d. Differential #3 - diff from the 13 days old to the 12 days old version
...
Every day after a new differential is taken, (b) and (c) would be
merged, keeping the number of differentials constant.
Not at the moment. Mykola has an old
work-in-progress PR which extends
"rbd import-diff" command to make this possible [1].
I didn't know this PR. Thank you very much.I'll evaluate this PR later.
Since you as
a user expected "rbd merge-diff" to be able to this, I wonder if this
functionality might be better placed under "rbd merge-diff"? That way
the operations on files would be separated from the operations on RBD
images.
Yes, I think so too.
For your backup system, couldn't you just merge the two oldest
differentials with "rbd merge-diff" instead? Instead of advancing the
full export, you would be advancing the first differential -- it would
represent a diff between the initial export and the "2 weeks" backup
over time.
Yes, it's possible. It's one of a workaround I thought. Then the
backup data are as follows:
a. The full backup taken at least 14 days ago.
b. The latest 14 days backup data
In this case, I concern the effect if (a) becomes too old. However, it
might be a groundless fear.
I would suggest re-capturing the full backup from time to time, just as
a precaution against something going wrong with a backup based on a too
long series of (merged) differentials. It might be groundless concern,
but then you can't be too careful when it comes to backups.
Thanks,
Ilya
18 Dec
18 Dec
3:14 a.m.
Hi Ilya,
Thank you for your advice!
Yes, it's
possible. It's one of a workaround I thought. Then the
backup data are as follows:
a. The full backup taken at least 14 days ago.
b. The latest 14 days backup data
I think it would be:
a. A full backup (taken potentially months ago, exact age doesn't
really matter)
b. Differential #1 - diff from the full backup to the 14 days old version
c. Differential #2 - diff from the 14 days old to the 13 days old version
d. Differential #3 - diff from the 13 days old to the 12 days old version
...
Every day after a new differential is taken, (b) and (c) would be
merged, keeping the number of differentials constant.
In this case, I concern the effect if (a) becomes too old. However, it
might be a groundless fear.
I would suggest re-capturing the full backup from time to time, just as
a precaution against something going wrong with a backup based on a too
long series of (merged) differentials. It might be groundless concern,
but then you can't be too careful when it comes to backups.
12 Jan
12 Jan
4:02 a.m.
Hi Ilya,
2023年12月18日(月) 9:14 Satoru Takeuchi <satoru.takeuchi(a)gmail.com>om>:
Hi Ilya,
> > Yes, it's possible. It's one of a workaround I thought. Then the
> > backup data are as follows:
> >
> > a. The full backup taken at least 14 days ago.
> > b. The latest 14 days backup data
>
> I think it would be:
>
> a. A full backup (taken potentially months ago, exact age doesn't
> really matter)
> b. Differential #1 - diff from the full backup to the 14 days old version
> c. Differential #2 - diff from the 14 days old to the 13 days old version
> d. Differential #3 - diff from the 13 days old to the 12 days old version
> ...
>
> Every day after a new differential is taken, (b) and (c) would be
> merged, keeping the number of differentials constant.
>
> >
> > In this case, I concern the effect if (a) becomes too old. However, it
> > might be a groundless fear.
>
> I would suggest re-capturing the full backup from time to time, just as
> a precaution against something going wrong with a backup based on a too
> long series of (merged) differentials. It might be groundless concern,
> but then you can't be too careful when it comes to backups.
I also got an advice from MykolaI and succeeded to merge the full
backup with the oldest diff.
The key point is capturing the full backup as a diff data.
My verification is:
1. Create an empty RBD image.
2. Update this image.
3. Capture a full backup F, by `rbd export-diff`.
4. Update this image again.
5. Capture a diff D by `rbd export-diff`
6. Create a new image which size is the same as an RBD image created at step1.
7. Import F by `rbd import-diff` and the contents is correct.
8. Import D by `rbd import-diff` and the contents is correct.
9. Merge F and D by `rbd merge-diff` and create D2.
10. Create an another new image.
11. Import D2 by `rbd import-diff` and the contents is as expected.
Thanks,
Satoru
128
days inactive
159
days old
5 comments
2 participants
participants (2)
-
Ilya Dryomov -
Satoru Takeuchi