On 01/05/2024 16:12, Satoru Takeuchi wrote:
I confirmed that incomplete data is left on `rbd import-diff` failure. I guess that this data is the part of snapshot. Could someone answer me the following questions?
Q1. Is it safe to use the RBD image (e.g. client I/O and snapshot management) even though incomplete data exists? Q2. Is there any way to clean up the incomplete data?
I read the following document and understand that this problem will be resolved after running `rbd import-diff` again.
https://ceph.io/en/news/blog/2013/incremental-snapshots-with-rbd/
Since overwriting the same data is idempotent, it’s safe to have an import-diff interrupted in the middle. However, it's difficult if I can't access the exported backup data anymore. For instance, I'm afraid of the following scenario.
1. Send the backup data from one DC (DC0) to another DC (DC1) periodically. 2. The backup data is created in DC0 and is sent directly to DC1 without persist backup data as a file. 3. Major power outage happens in DC0 and it's impossible to re-generate the backup data for a long time.
I simulated this problem as follows:
1. Create an RBD image. 2. Write some data to this image. 3. Create a snapshot S0. 4. Write another data to this image. 5. Create a snapshot S1. 6. Create a backup data consists of the difference between S0 and S1 by running rbd export-diff. 7. Delete the last byte of the backup data, which is 'e' and means the end of the backup data, to inject import-diff failure. 8. Delete S1. 9. Run rbd import-diff to apply the broken backup data created in the step 7.
Then step9 failed and S1 was not created. However, the number of RADOS objects and the storage usage has increased.
before ``` $ rados -p replicapool df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR replicapool 11 MiB 24 9 24 0 0 0 3609 53 MiB 279 41 MiB 0 B 0 B
total_objects 24 total_used 39 MiB total_avail 32 GiB total_space 32 GiB ```
after: ``` $ rados -p replicapool df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR replicapool 12 MiB 25 9 25 0 0 0 3531 53 MiB 278 41 MiB 0 B 0 B
total_objects 25 total_used 40 MiB total_avail 32 GiB total_space 32 GiB ```
The incomplete data seem to increase if rbd import-diff fails again and again. The following output was get after the above-mentioned step9 100 times.
``` $ rados -p replicapool df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR replicapool 12 MiB 25 9 25 0 0 0 7925 104 MiB 1308 164 MiB 0 B 0 B
total_objects 25 total_used 58 MiB total_avail 32 GiB total_space 32 GiB ```
Thanks, Satoru _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
the image is not in a consistent state so should not be used as is. if you no longer have access to the source image or its exported data, you should be able to use the rbd snap rollback command to rollback the destination image to its last known good snapshot, the destination snapshots get created from the import-diff command with names matching source snapshots. /maged