Ceph's ino32 mount option has trivial collisions. The hash is `ceph_ino_to_ino32` here: https://github.com/torvalds/linux/blob/master/fs/ceph/super.h#L438 A simple collision can be demonstrated: def ceph_ino_to_ino32(vino): ino = vino & 0xffffffff; ino ^= vino >> 32; if not ino: ino = 2 return ino print ceph_ino_to_ino32(0x10000000301) # 0x302'nd inode on mds.0 print ceph_ino_to_ino32(0x20000000001) # 2nd inode on mds.1 Both 0x10000000301 and 0x20000000001 hash to ino32=513 So I know that collisions are very likely when using multiple active MDSs. So I wondered: if we pin the mount prefix to a single mds, then maybe collisions are less likely? Seems so, but I still found exactly one collision in the range(1<<40) to (1<<40)+(1<<25) : 0x10000000102 and 0x10000000100 both hash to ino32=2 Since the collisions are inevitable -- are they handled in some sane/safe way on the mds side? If not -- maybe we should improve or remove the ino32 kernel option? Cheers, Dan On Wed, Oct 16, 2019 at 9:48 AM Ingo Schmidt <i.schmidt@langeoog.de> wrote:
This is not quite true. The numberspace of MD5 is much greater than 2³², (2¹²⁸ exactly) and as long as you don't exhaust this Numberspace, the probability of having a collision is roughly equally likely as with any other input. There might be collisions, and the more Data you have, i.e. the more Adresses you use, the higher the probability. Security researchers have shown that it is possible to create collisions, but it is very rare.
I cannot give you an estimate of the consequences of a collision though. It's a matter of what data is stored at that address and how Programs/OSes and even Ceph deal with this. I would suspect, ceph would find a checksum mismatch upon scrubbing. But I don't know how, or if ceph could or would correct this, as the two addresses with the same MD5sum have equally valid copies and i think in such a case it is undecidable, which Data is correct.
Greetings Ingo
----- Ursprüngliche Mail ----- Von: "Nathan Fish" <lordcirth@gmail.com> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Dienstag, 15. Oktober 2019 19:40:05 Betreff: [ceph-users] Re: CephFS and 32-bit Inode Numbers
I'm not sure exactly what would happen on an inode collision, but I'm guessing Bad Things. If my math is correct, a 2^32 inode space will have roughly 1 collision per 2^16 entries. As that's only 65536, that's not safe at all.
On Mon, Oct 14, 2019 at 8:14 AM Dan van der Ster <dan@vanderster.com> wrote:
OK I found that the kernel has an "ino32" mount option which hashes 64 bit inos to 32-bit space. Has anyone tried this? What happens if two files collide?
-- Dan
On Mon, Oct 14, 2019 at 1:18 PM Dan van der Ster <dan@vanderster.com> wrote:
Hi all,
One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3.
I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3.
Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?)
Thanks!
Dan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io