Hi all, One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3. I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3. Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?) Thanks! Dan
OK I found that the kernel has an "ino32" mount option which hashes 64 bit inos to 32-bit space. Has anyone tried this? What happens if two files collide? -- Dan On Mon, Oct 14, 2019 at 1:18 PM Dan van der Ster <dan@vanderster.com> wrote:
Hi all,
One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3.
I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3.
Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?)
Thanks!
Dan
I'm not sure exactly what would happen on an inode collision, but I'm guessing Bad Things. If my math is correct, a 2^32 inode space will have roughly 1 collision per 2^16 entries. As that's only 65536, that's not safe at all. On Mon, Oct 14, 2019 at 8:14 AM Dan van der Ster <dan@vanderster.com> wrote:
OK I found that the kernel has an "ino32" mount option which hashes 64 bit inos to 32-bit space. Has anyone tried this? What happens if two files collide?
-- Dan
On Mon, Oct 14, 2019 at 1:18 PM Dan van der Ster <dan@vanderster.com> wrote:
Hi all,
One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3.
I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3.
Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?)
Thanks!
Dan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Den tis 15 okt. 2019 kl 19:40 skrev Nathan Fish <lordcirth@gmail.com>:
I'm not sure exactly what would happen on an inode collision, but I'm guessing Bad Things. If my math is correct, a 2^32 inode space will have roughly 1 collision per 2^16 entries. As that's only 65536, that's not safe at all.
Yeah, the birthday paradox will make sure you hit it very soon. 8-( -- May the most significant bit of your life be positive.
Once upon a time ceph-fuse did its own internal hash-map of live inodes to handle that (by just remembering which 64-bit inode any 32-bit one actually referred to). Unfortunately I believe this has been ripped out because it caused problems when the kernel tried to do lookups on 32-bit inodes that were so old they'd been recycled or dropped out of the mapping table. It's conceivable the kernel can implement this more safely and I'd certainly defer to Zheng or somebody as this is just out of my head and from prepping the source tree to see what still exists, but I would not expect it to be a safe long-term solution. -Greg On Tue, Oct 15, 2019 at 12:01 PM Janne Johansson <icepic.dz@gmail.com> wrote:
Den tis 15 okt. 2019 kl 19:40 skrev Nathan Fish <lordcirth@gmail.com>:
I'm not sure exactly what would happen on an inode collision, but I'm guessing Bad Things. If my math is correct, a 2^32 inode space will have roughly 1 collision per 2^16 entries. As that's only 65536, that's not safe at all.
Yeah, the birthday paradox will make sure you hit it very soon. 8-(
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This is not quite true. The numberspace of MD5 is much greater than 2³², (2¹²⁸ exactly) and as long as you don't exhaust this Numberspace, the probability of having a collision is roughly equally likely as with any other input. There might be collisions, and the more Data you have, i.e. the more Adresses you use, the higher the probability. Security researchers have shown that it is possible to create collisions, but it is very rare. I cannot give you an estimate of the consequences of a collision though. It's a matter of what data is stored at that address and how Programs/OSes and even Ceph deal with this. I would suspect, ceph would find a checksum mismatch upon scrubbing. But I don't know how, or if ceph could or would correct this, as the two addresses with the same MD5sum have equally valid copies and i think in such a case it is undecidable, which Data is correct. Greetings Ingo ----- Ursprüngliche Mail ----- Von: "Nathan Fish" <lordcirth@gmail.com> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Dienstag, 15. Oktober 2019 19:40:05 Betreff: [ceph-users] Re: CephFS and 32-bit Inode Numbers I'm not sure exactly what would happen on an inode collision, but I'm guessing Bad Things. If my math is correct, a 2^32 inode space will have roughly 1 collision per 2^16 entries. As that's only 65536, that's not safe at all. On Mon, Oct 14, 2019 at 8:14 AM Dan van der Ster <dan@vanderster.com> wrote:
OK I found that the kernel has an "ino32" mount option which hashes 64 bit inos to 32-bit space. Has anyone tried this? What happens if two files collide?
-- Dan
On Mon, Oct 14, 2019 at 1:18 PM Dan van der Ster <dan@vanderster.com> wrote:
Hi all,
One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3.
I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3.
Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?)
Thanks!
Dan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ceph's ino32 mount option has trivial collisions. The hash is `ceph_ino_to_ino32` here: https://github.com/torvalds/linux/blob/master/fs/ceph/super.h#L438 A simple collision can be demonstrated: def ceph_ino_to_ino32(vino): ino = vino & 0xffffffff; ino ^= vino >> 32; if not ino: ino = 2 return ino print ceph_ino_to_ino32(0x10000000301) # 0x302'nd inode on mds.0 print ceph_ino_to_ino32(0x20000000001) # 2nd inode on mds.1 Both 0x10000000301 and 0x20000000001 hash to ino32=513 So I know that collisions are very likely when using multiple active MDSs. So I wondered: if we pin the mount prefix to a single mds, then maybe collisions are less likely? Seems so, but I still found exactly one collision in the range(1<<40) to (1<<40)+(1<<25) : 0x10000000102 and 0x10000000100 both hash to ino32=2 Since the collisions are inevitable -- are they handled in some sane/safe way on the mds side? If not -- maybe we should improve or remove the ino32 kernel option? Cheers, Dan On Wed, Oct 16, 2019 at 9:48 AM Ingo Schmidt <i.schmidt@langeoog.de> wrote:
This is not quite true. The numberspace of MD5 is much greater than 2³², (2¹²⁸ exactly) and as long as you don't exhaust this Numberspace, the probability of having a collision is roughly equally likely as with any other input. There might be collisions, and the more Data you have, i.e. the more Adresses you use, the higher the probability. Security researchers have shown that it is possible to create collisions, but it is very rare.
I cannot give you an estimate of the consequences of a collision though. It's a matter of what data is stored at that address and how Programs/OSes and even Ceph deal with this. I would suspect, ceph would find a checksum mismatch upon scrubbing. But I don't know how, or if ceph could or would correct this, as the two addresses with the same MD5sum have equally valid copies and i think in such a case it is undecidable, which Data is correct.
Greetings Ingo
----- Ursprüngliche Mail ----- Von: "Nathan Fish" <lordcirth@gmail.com> An: "ceph-users" <ceph-users@ceph.io> Gesendet: Dienstag, 15. Oktober 2019 19:40:05 Betreff: [ceph-users] Re: CephFS and 32-bit Inode Numbers
I'm not sure exactly what would happen on an inode collision, but I'm guessing Bad Things. If my math is correct, a 2^32 inode space will have roughly 1 collision per 2^16 entries. As that's only 65536, that's not safe at all.
On Mon, Oct 14, 2019 at 8:14 AM Dan van der Ster <dan@vanderster.com> wrote:
OK I found that the kernel has an "ino32" mount option which hashes 64 bit inos to 32-bit space. Has anyone tried this? What happens if two files collide?
-- Dan
On Mon, Oct 14, 2019 at 1:18 PM Dan van der Ster <dan@vanderster.com> wrote:
Hi all,
One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3.
I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3.
Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?)
Thanks!
Dan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Mon, Oct 14, 2019 at 7:19 PM Dan van der Ster <dan@vanderster.com> wrote:
Hi all,
One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3.
I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3.
Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?)
ceph-fuse has client_use_faked_inos option. When it is enabled, ceph-fuse maps 64bits inode numbers to 32bits. It works as long as client has less than 2^32 inodes cached. So far there is no kernel client counterpart. Regards Yan, Zheng
Thanks!
Dan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Does your 32-bit application actually use the inode numbers? Or is it just trying to read other metadata (such as filenames in a directory, file sizes, etc)? If it's the latter, you could use LD_PRELOAD to wrap the calls and return fake/mangled inode numbers (since the application doesn't care about them anyway). See here for more details: https://www.tcm.phy.cam.ac.uk/sw/inodes64.html -----Original Message----- From: Yan, Zheng <ukernel@gmail.com> Sent: Wednesday, October 16, 2019 8:07 PM To: Dan van der Ster <dan@vanderster.com> Cc: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Re: CephFS and 32-bit Inode Numbers On Mon, Oct 14, 2019 at 7:19 PM Dan van der Ster <dan@vanderster.com> wrote:
Hi all,
One of our users has some 32-bit commercial software that they want to use with CephFS, but it's not working because our inode numbers are too large. E.g. his application gets a "file too big" error trying to stat inode 0x40008445FB3.
I'm aware that CephFS is offsets the inode numbers by (mds_rank + 1) * 2^40; in the case above the file is managed by mds.3.
Did anyone see this same issue and find a workaround? (I read that GlusterFS has an enable-in32 client option -- does CephFS have something like that planned?)
ceph-fuse has client_use_faked_inos option. When it is enabled, ceph-fuse maps 64bits inode numbers to 32bits. It works as long as client has less than 2^32 inodes cached. So far there is no kernel client counterpart. Regards Yan, Zheng
Thanks!
Dan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (7)
-
Dan van der Ster
-
Darrell Enns
-
Gregory Farnum
-
Ingo Schmidt
-
Janne Johansson
-
Nathan Fish
-
Yan, Zheng