Multisite RGW data corruption (not 14.2.1 curl issue)
Hello, I have setup two separate Ceph clusters with RGW instance each and trying to achieve multisite data synchronization. Primary runs 13.2.5, slave runs 14.2.2 (I have upgraded slave side from 14.2.1 due to known data corruption during transfer due to curl errors). I have emptied slave zone and allowed sync to run from beginning to the end. Then I have recalculated MD5 hashes over original data and over data in slave zones and found that in some cases they do not match. Data corruption is evident. Comparison of data byte for byte shows that some parts of data just moved around (for example file has correct bytes from 0 to 513k and then exactly the same bytes from position of 513k up to 1026k - feels like as some sort of buffer issue). File size is correct. I have read the RGW sources and could not find anything what can cause such sort of behavior however I could not find one single piece of code which I would consider critical: during FetchRemote RGW obtains object's etag from remote RGW but apparently there nothing what recalculates MD5 from actual data and comparing it to etag received to ensure that data was correctly transferred. Even broken data then stored to local cluster and into bucket index. Then there just nothing further to prevent wrong data to reach end user downloading from slave zone. Such check should happen regardless of sync backend in use: objects with broken data should not be stored in bucket index at all. No one needs broken data and it is better just to fail object sync to allow it to try again later instead of storing faulty data and then allowing end users to download it. Is it me who cannot see such sort of check or it is actually missing?! And if it is not there at all I think it should be quite high on TODO list. Regards, Vladimir
Yes there is no checksum check in RadosSync at this stage... we discussed a bit around it when handling the curl issue. The challenge is for multipart object, the e-tag is not the checksum of the object itself, instead , it is the checksum of the manifest. Special (internal) API is needed to expose the manifest from src zone to slave zone. <vladimir@bashkirtsev.com> 于2019年8月20日周二 上午12:24写道：
Hello,
I have setup two separate Ceph clusters with RGW instance each and trying to achieve multisite data synchronization. Primary runs 13.2.5, slave runs 14.2.2 (I have upgraded slave side from 14.2.1 due to known data corruption during transfer due to curl errors). I have emptied slave zone and allowed sync to run from beginning to the end. Then I have recalculated MD5 hashes over original data and over data in slave zones and found that in some cases they do not match. Data corruption is evident.
Comparison of data byte for byte shows that some parts of data just moved around (for example file has correct bytes from 0 to 513k and then exactly the same bytes from position of 513k up to 1026k - feels like as some sort of buffer issue). File size is correct. I have read the RGW sources and could not find anything what can cause such sort of behavior however I could not find one single piece of code which I would consider critical: during FetchRemote RGW obtains object's etag from remote RGW but apparently there nothing what recalculates MD5 from actual data and comparing it to etag received to ensure that data was correctly transferred. Even broken data then stored to local cluster and into bucket index. Then there just nothing further to prevent wrong data to reach end user downloading from slave zone.
Such check should happen regardless of sync backend in use: objects with broken data should not be stored in bucket index at all. No one needs broken data and it is better just to fail object sync to allow it to try again later instead of storing faulty data and then allowing end users to download it.
Is it me who cannot see such sort of check or it is actually missing?! And if it is not there at all I think it should be quite high on TODO list.
Regards, Vladimir _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That's a bit disappointing to say the least... It pretty much renders RGW unusable for production use at the moment then. Do you know how much work and when manifest checksumming is likely to happen? Should I hold my breath? In the mean time then perhaps I should remove corrupted objects from slave zone manually but I seem not be able to find a way to do it. Any ideas how to remove a single object from slave zone?
participants (2)
-
vladimir＠bashkirtsev.com
-
Xiaoxi Chen