Hi all,
I have a problem with exporting 2 different sub-folder ceph-fs kernel mounts via nfsd to
the same IP address. The top-level structure on the ceph fs is something like /A/S1 and
/A/S2. On a file server I mount /A/S1 and /A/S2 as two different file systems under
/mnt/S1 and /mnt/S2 using the ceph fs kernel client. Then, these 2 mounts are exported
with lines like these in /etc/exports:
/mnt/S1 -options NET
/mnt/S2 -options IP
IP is an element of NET, meaning that the host at IP should be the only host being able to
access /mnt/S1 and /mnt/S2. What we observe is that any attempt to mount the export
/mnt/S1 on the host at IP results in /mnt/S2 being mounted instead.
My first guess was that here we have a clash of fsids and the ceph fs is simply reporting
the same fsid to nfsd and, hence, nfsd thinks both mountpoints contain the same. So I
modified the second export line to
/mnt/S2 -options,fsid=100 IP
to no avail. The two folders are completely disjoint, neither symlinks nor hard-links
between them. So it should be safe to export these as 2 different file systems.
Exporting such constructs to non-overlapping networks/IPs works as expected - even when
exporting subdirs of a dir (like exporting /A/B and /A/B/C from the same file server to
strictly different IPs). It seems the same-IP config that breaks expectations.
Am I missing here a magic -yes-i-really-know-what-i-am-doing hack? The file server is on
AlmaLinux release 8.7 (Stone Smilodon) and all ceph packages match the ceph version
octopus latest of our cluster.
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
Show replies by date
To answer my own question, assigning an explicit fsid to only one of the exports seems to
overwrite the default for all file systems with the same default fsid. Hence, the export
definitions
/mnt/S1 -options NET
/mnt/S2 -options IP
and
/mnt/S1 -options NET
/mnt/S2 -options,fsid=100 IP
are equivalent. My expectation was that /mnt/S1 would retain the default fsid. Using
explicit fsids for each export as in:
/mnt/S1 -options,fsid=100 NET
/mnt/S2 -options,fsid=101 IP
leads to the desired result.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Frank Schilder <frans(a)dtu.dk>
Sent: Monday, May 15, 2023 6:51 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] NFS export of 2 disjoint sub-dir mounts
Hi all,
I have a problem with exporting 2 different sub-folder ceph-fs kernel mounts via nfsd to
the same IP address. The top-level structure on the ceph fs is something like /A/S1 and
/A/S2. On a file server I mount /A/S1 and /A/S2 as two different file systems under
/mnt/S1 and /mnt/S2 using the ceph fs kernel client. Then, these 2 mounts are exported
with lines like these in /etc/exports:
/mnt/S1 -options NET
/mnt/S2 -options IP
IP is an element of NET, meaning that the host at IP should be the only host being able to
access /mnt/S1 and /mnt/S2. What we observe is that any attempt to mount the export
/mnt/S1 on the host at IP results in /mnt/S2 being mounted instead.
My first guess was that here we have a clash of fsids and the ceph fs is simply reporting
the same fsid to nfsd and, hence, nfsd thinks both mountpoints contain the same. So I
modified the second export line to
/mnt/S2 -options,fsid=100 IP
to no avail. The two folders are completely disjoint, neither symlinks nor hard-links
between them. So it should be safe to export these as 2 different file systems.
Exporting such constructs to non-overlapping networks/IPs works as expected - even when
exporting subdirs of a dir (like exporting /A/B and /A/B/C from the same file server to
strictly different IPs). It seems the same-IP config that breaks expectations.
Am I missing here a magic -yes-i-really-know-what-i-am-doing hack? The file server is on
AlmaLinux release 8.7 (Stone Smilodon) and all ceph packages match the ceph version
octopus latest of our cluster.
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io