Is there any way to merge an rbd image's full backup and a diff?
Hi, I'm developing RBD images' backup system. In my case, a backup data must be stored at least two weeks. To meet this requirement, I'd like to take backups as follows: 1. Take a full backup by rbd export first. 2. Take a differencial backups everyday. 3. Merge the full backup and the oldest (taken two weeks ago) diff. As a result of evaluation, I confirmed there is no problem in step 1 and 2. However, I found that step 3 couldn't be accomplished by `rbd merge-diff <full backup> <diff>` because `rbd merge-diff` only accepts a diff as a first parameter. Is there any way to merge a full backup and a diff? Thanks, Satoru
On Tue, Dec 12, 2023 at 1:03 AM Satoru Takeuchi <satoru.takeuchi@gmail.com> wrote:
Hi,
I'm developing RBD images' backup system. In my case, a backup data must be stored at least two weeks. To meet this requirement, I'd like to take backups as follows:
1. Take a full backup by rbd export first. 2. Take a differencial backups everyday. 3. Merge the full backup and the oldest (taken two weeks ago) diff.
As a result of evaluation, I confirmed there is no problem in step 1 and 2. However, I found that step 3 couldn't be accomplished by `rbd merge-diff <full backup> <diff>` because `rbd merge-diff` only accepts a diff as a first parameter. Is there any way to merge a full backup and a diff?
Hi Satoru, Not at the moment. Mykola has an old work-in-progress PR which extends "rbd import-diff" command to make this possible [1]. Since you as a user expected "rbd merge-diff" to be able to this, I wonder if this functionality might be better placed under "rbd merge-diff"? That way the operations on files would be separated from the operations on RBD images. For your backup system, couldn't you just merge the two oldest differentials with "rbd merge-diff" instead? Instead of advancing the full export, you would be advancing the first differential -- it would represent a diff between the initial export and the "2 weeks" backup over time. [1] https://github.com/ceph/ceph/pull/41375 Thanks, Ilya
Hi Ilya, 2023年12月12日(火) 21:23 Ilya Dryomov <idryomov@gmail.com>:
Not at the moment. Mykola has an old work-in-progress PR which extends "rbd import-diff" command to make this possible [1].
I didn't know this PR. Thank you very much.I'll evaluate this PR later.
Since you as a user expected "rbd merge-diff" to be able to this, I wonder if this functionality might be better placed under "rbd merge-diff"? That way the operations on files would be separated from the operations on RBD images.
Yes, I think so too.
For your backup system, couldn't you just merge the two oldest differentials with "rbd merge-diff" instead? Instead of advancing the full export, you would be advancing the first differential -- it would represent a diff between the initial export and the "2 weeks" backup over time.
Yes, it's possible. It's one of a workaround I thought. Then the backup data are as follows: a. The full backup taken at least 14 days ago. b. The latest 14 days backup data In this case, I concern the effect if (a) becomes too old. However, it might be a groundless fear. I'll give it some more thought. Thank you again. Best, Satoru
On Wed, Dec 13, 2023 at 12:48 AM Satoru Takeuchi <satoru.takeuchi@gmail.com> wrote:
Hi Ilya,
2023年12月12日(火) 21:23 Ilya Dryomov <idryomov@gmail.com>:
Not at the moment. Mykola has an old work-in-progress PR which extends "rbd import-diff" command to make this possible [1].
I didn't know this PR. Thank you very much.I'll evaluate this PR later.
Since you as a user expected "rbd merge-diff" to be able to this, I wonder if this functionality might be better placed under "rbd merge-diff"? That way the operations on files would be separated from the operations on RBD images.
Yes, I think so too.
For your backup system, couldn't you just merge the two oldest differentials with "rbd merge-diff" instead? Instead of advancing the full export, you would be advancing the first differential -- it would represent a diff between the initial export and the "2 weeks" backup over time.
Yes, it's possible. It's one of a workaround I thought. Then the backup data are as follows:
a. The full backup taken at least 14 days ago. b. The latest 14 days backup data
I think it would be: a. A full backup (taken potentially months ago, exact age doesn't really matter) b. Differential #1 - diff from the full backup to the 14 days old version c. Differential #2 - diff from the 14 days old to the 13 days old version d. Differential #3 - diff from the 13 days old to the 12 days old version ... Every day after a new differential is taken, (b) and (c) would be merged, keeping the number of differentials constant.
In this case, I concern the effect if (a) becomes too old. However, it might be a groundless fear.
I would suggest re-capturing the full backup from time to time, just as a precaution against something going wrong with a backup based on a too long series of (merged) differentials. It might be groundless concern, but then you can't be too careful when it comes to backups. Thanks, Ilya
Hi Ilya,
Yes, it's possible. It's one of a workaround I thought. Then the backup data are as follows:
a. The full backup taken at least 14 days ago. b. The latest 14 days backup data
I think it would be:
a. A full backup (taken potentially months ago, exact age doesn't really matter) b. Differential #1 - diff from the full backup to the 14 days old version c. Differential #2 - diff from the 14 days old to the 13 days old version d. Differential #3 - diff from the 13 days old to the 12 days old version ...
Every day after a new differential is taken, (b) and (c) would be merged, keeping the number of differentials constant.
In this case, I concern the effect if (a) becomes too old. However, it might be a groundless fear.
I would suggest re-capturing the full backup from time to time, just as a precaution against something going wrong with a backup based on a too long series of (merged) differentials. It might be groundless concern, but then you can't be too careful when it comes to backups.
Thank you for your advice!
Hi Ilya, 2023年12月18日(月) 9:14 Satoru Takeuchi <satoru.takeuchi@gmail.com>:
Hi Ilya,
Yes, it's possible. It's one of a workaround I thought. Then the backup data are as follows:
a. The full backup taken at least 14 days ago. b. The latest 14 days backup data
I think it would be:
a. A full backup (taken potentially months ago, exact age doesn't really matter) b. Differential #1 - diff from the full backup to the 14 days old version c. Differential #2 - diff from the 14 days old to the 13 days old version d. Differential #3 - diff from the 13 days old to the 12 days old version ...
Every day after a new differential is taken, (b) and (c) would be merged, keeping the number of differentials constant.
In this case, I concern the effect if (a) becomes too old. However, it might be a groundless fear.
I would suggest re-capturing the full backup from time to time, just as a precaution against something going wrong with a backup based on a too long series of (merged) differentials. It might be groundless concern, but then you can't be too careful when it comes to backups.
I also got an advice from MykolaI and succeeded to merge the full backup with the oldest diff. The key point is capturing the full backup as a diff data. My verification is: 1. Create an empty RBD image. 2. Update this image. 3. Capture a full backup F, by `rbd export-diff`. 4. Update this image again. 5. Capture a diff D by `rbd export-diff` 6. Create a new image which size is the same as an RBD image created at step1. 7. Import F by `rbd import-diff` and the contents is correct. 8. Import D by `rbd import-diff` and the contents is correct. 9. Merge F and D by `rbd merge-diff` and create D2. 10. Create an another new image. 11. Import D2 by `rbd import-diff` and the contents is as expected. Thanks, Satoru
participants (2)
-
Ilya Dryomov
-
Satoru Takeuchi