consistency of import-diff
Hello, does anybody know whether there is any mechanism to make sure an image looks like the original after an import-diff? While doing ceph backups on another ceph cluster i currently do a fresh import every 7 days. So i'm sure if something went wrong with import-diff i have a fresh one every 7 days. Otherwise i waste a lot of backup storage. So i wanted to know if there is any way to be sure that the image is OK and save and match the orignal snapshot afterwards. Greets, Stefan
Hi, You can use a full local export, piped to some hash program (this is what Backurne¹ does) : rbd export <image> - | xxhsum Then, check the hash consistency with the original Regards, [1] https://github.com/JackSlateur/backurne On 3/3/20 8:46 PM, Stefan Priebe - Profihost AG wrote:
Hello,
does anybody know whether there is any mechanism to make sure an image looks like the original after an import-diff?
While doing ceph backups on another ceph cluster i currently do a fresh import every 7 days. So i'm sure if something went wrong with import-diff i have a fresh one every 7 days.
Otherwise i waste a lot of backup storage. So i wanted to know if there is any way to be sure that the image is OK and save and match the orignal snapshot afterwards.
Greets, Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Am 03.03.20 um 20:54 schrieb Jack:
Hi,
You can use a full local export, piped to some hash program (this is what Backurne¹ does) : rbd export <image> - | xxhsum Then, check the hash consistency with the original
Thanks for the suggestion but this still needs to run an rbd export on the source and target snapshot everytime to compare hashes? Which is slow if you talk about 100's of terrabytes of data isn't it? Stefan
Regards,
[1] https://github.com/JackSlateur/backurne
On 3/3/20 8:46 PM, Stefan Priebe - Profihost AG wrote:
Hello,
does anybody know whether there is any mechanism to make sure an image looks like the original after an import-diff?
While doing ceph backups on another ceph cluster i currently do a fresh import every 7 days. So i'm sure if something went wrong with import-diff i have a fresh one every 7 days.
Otherwise i waste a lot of backup storage. So i wanted to know if there is any way to be sure that the image is OK and save and match the orignal snapshot afterwards.
Greets, Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den tis 3 mars 2020 kl 21:48 skrev Stefan Priebe - Profihost AG < s.priebe@profihost.ag>:
You can use a full local export, piped to some hash program (this is what Backurne¹ does) : rbd export <image> - | xxhsum Then, check the hash consistency with the original
Thanks for the suggestion but this still needs to run an rbd export on the source and target snapshot everytime to compare hashes? Which is slow if you talk about 100's of terrabytes of data isn't it?
Sorry for not adding anything to solve your issue, but wouldn't *any* method to validate that 100TB is identical to some other 100TB always be slow? It seems slightly illogical to me to mistrust that copy A is 100% identical to copy B after some kind of sync/replication/snap/rebuild but then hope for a method which doesn't involve reading 100s of TB to make sure they actually are. -- May the most significant bit of your life be positive.
On 3/3/20 8:46 PM, Stefan Priebe - Profihost AG wrote:
Hello,
does anybody know whether there is any mechanism to make sure an image looks like the original after an import-diff?
While doing ceph backups on another ceph cluster i currently do a fresh import every 7 days. So i'm sure if something went wrong with import-diff i have a fresh one every 7 days.
Otherwise i waste a lot of backup storage. So i wanted to know if there is any way to be sure that the image is OK and save and match the orignal snapshot afterwards.
But how can you be sure that the program that verifies this for you doesn't have a bug? I understand what you are trying to do, but it's a trade-off. Endless snapshots are also a danger because bit-rot can sneak in somewhere which you might not notice. A fresh export (full copy) every X period protects you against this. Wido
Greets, Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2020-03-04T15:44:34, Wido den Hollander <wido@42on.com> wrote:
I understand what you are trying to do, but it's a trade-off. Endless snapshots are also a danger because bit-rot can sneak in somewhere which you might not notice.
A fresh export (full copy) every X period protects you against this.
Hrm. We have checksums on the actual OSD data, so it ought to be possible to add these to the export/import/diff bits so it can be verified faster. (Well, barring bugs.) -- SUSE Software Solutions Germany GmbH, MD: Felix Imendörffer, HRB 36809 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
On 3/4/20 3:49 PM, Lars Marowsky-Bree wrote:
On 2020-03-04T15:44:34, Wido den Hollander <wido@42on.com> wrote:
I understand what you are trying to do, but it's a trade-off. Endless snapshots are also a danger because bit-rot can sneak in somewhere which you might not notice.
A fresh export (full copy) every X period protects you against this.
Hrm. We have checksums on the actual OSD data, so it ought to be possible to add these to the export/import/diff bits so it can be verified faster.
(Well, barring bugs.)
I mainly meant bugs, I should have clarified that better. Do you trust the technology you want to backup to create the proper backup for you? With that I mean, what if librbd or librados contains a bug which corrupts all your backups? You think the backups all went fine because the snapshots seem consistent on both ends, but you are not sure until you actually test a restore. Those are the things I take into consideration when using such technologies. Wido
Am 04.03.20 um 16:02 schrieb Wido den Hollander:
On 3/4/20 3:49 PM, Lars Marowsky-Bree wrote:
On 2020-03-04T15:44:34, Wido den Hollander <wido@42on.com> wrote:
I understand what you are trying to do, but it's a trade-off. Endless snapshots are also a danger because bit-rot can sneak in somewhere which you might not notice.
A fresh export (full copy) every X period protects you against this.
Hrm. We have checksums on the actual OSD data, so it ought to be possible to add these to the export/import/diff bits so it can be verified faster.
(Well, barring bugs.)
I mainly meant bugs, I should have clarified that better.
Do you trust the technology you want to backup to create the proper backup for you? With that I mean, what if librbd or librados contains a bug which corrupts all your backups?
You think the backups all went fine because the snapshots seem consistent on both ends, but you are not sure until you actually test a restore.
Yes and know. If the object checksums inside ceph are equal it must be a really bad bug. Sure this can happen but i think the chances are very low Stefan
Those are the things I take into consideration when using such technologies.
Wido
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den tors 5 mars 2020 kl 08:13 skrev Stefan Priebe - Profihost AG < s.priebe@profihost.ag>:
Hrm. We have checksums on the actual OSD data, so it ought to be possible to add these to the export/import/diff bits so it can be verified faster. (Well, barring bugs.)
I mainly meant bugs, I should have clarified that better.
Do you trust the technology you want to backup to create the proper backup for you? With that I mean, what if librbd or librados contains a bug which corrupts all your backups?
You think the backups all went fine because the snapshots seem consistent on both ends, but you are not sure until you actually test a restore.
Yes and know. If the object checksums inside ceph are equal it must be a really bad bug. Sure this can happen but i think the chances are very low
Are we talking about checksums from months ago when it was created or the "current" data, which may or may not have been changed or not fully copied over to the destination in this case? It smells a bit like shuffling around the vocabulary in order to try not to end up in "dang, in order to actually know, some part must actually read 100TB data" which I think is where you have to go in order for you to move from "I think its ok" to "I know it is ok" after a copy/move/rebuild from A to B. I'm not trying to be obtuse or anything, just noting that when you sync something from A to B and ceph claims it is there, it has done as good as it can to check that the operation was done 100%. There might be reasons for distrusting this or not, but when you DO decide to mistrust, it feels weird to move backwards again and go "well, tcp checksums would have caught transmission errors, filesystem/OSD checksums should have caught storage errors and ..." because the idea if I understood it correctly to begin with was how to remove doubt about that all the various levels of operations have actually managed to create a perfect copy or not, not just list things that helped ceph make it probably-ok-but-I-dont-know to begin with. -- May the most significant bit of your life be positive.
Am 04.03.20 um 15:49 schrieb Lars Marowsky-Bree:
On 2020-03-04T15:44:34, Wido den Hollander <wido@42on.com> wrote:
I understand what you are trying to do, but it's a trade-off. Endless snapshots are also a danger because bit-rot can sneak in somewhere which you might not notice.
A fresh export (full copy) every X period protects you against this.
Hrm. We have checksums on the actual OSD data, so it ought to be possible to add these to the export/import/diff bits so it can be verified faster.
That was something i thought about.
(Well, barring bugs.)
Am 04.03.20 um 15:44 schrieb Wido den Hollander:
On 3/3/20 8:46 PM, Stefan Priebe - Profihost AG wrote:
Hello,
does anybody know whether there is any mechanism to make sure an image looks like the original after an import-diff?
While doing ceph backups on another ceph cluster i currently do a fresh import every 7 days. So i'm sure if something went wrong with import-diff i have a fresh one every 7 days.
Otherwise i waste a lot of backup storage. So i wanted to know if there is any way to be sure that the image is OK and save and match the orignal snapshot afterwards.
But how can you be sure that the program that verifies this for you doesn't have a bug?
May be with object checksums from ceph or something like this.
I understand what you are trying to do, but it's a trade-off. Endless snapshots are also a danger because bit-rot can sneak in somewhere which you might not notice.
A fresh export (full copy) every X period protects you against this.
Yes that is what i'm doing since years but i just wanted to know if there might be a better way.
Wido
Greets, Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Jack
-
Janne Johansson
-
Lars Marowsky-Bree
-
Stefan Priebe - Profihost AG
-
Wido den Hollander