ceph recipe for nfs exports
Hi you all, I'm almost new to ceph and I'm understanding, day by day, why the official support is so expansive :) I setting up a ceph nfs network cluster whose recipe can be found here below. ####################### --> cluster creation cephadm bootstrap --mon-ip 10.20.20.81 --cluster-network 10.20.20.0/24 --fsid $FSID --initial-dashboard-user adm \ --initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate --allow-fqdn-hostname --ssl-dashboard-port 443 \ --dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt --dashboard-key /etc/ssl/wildcard.it/wildcard.it.key \ --allow-overwrite --cleanup-on-failure cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring cephadm add-repo --release reef && cephadm install ceph-common --> adding hosts and set labels for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do ssh-copy-id -f -i /etc/ceph/ceph.pub root@$IP ; done ceph orch host add cephstage01 10.20.20.81 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage02 10.20.20.82 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage03 10.20.20.83 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstagedatanode01 10.20.20.84 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode02 10.20.20.85 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode03 10.20.20.86 --labels osd,nfs,prometheus --> network setup and daemons deploy ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24 ceph orch apply mon --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply mgr --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply prometheus --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply grafana --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply node-exporter ceph orch apply alertmanager ceph config set mgr mgr/cephadm/secure_monitoring_stack true --> disks and osd setup for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; do ssh root@$IP "hostname && wipefs -a -f /dev/sdb&& wipefs -a -f /dev/sdc"; done ceph config set mgr mgr/cephadm/device_enhanced_scan true for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device ls --hostname=$IP --wide --refresh ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdb; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdc ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdb ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdc ; done --> ganesha nfs cluster ceph mgr module enable nfs ceph fs volume create vol1 ceph nfs cluster create nfs-cephfs "cephstagedatanode01,cephstagedatanode02,cephstagedatanode03" --ingress --virtual-ip 192.168.7.80 --ingress-mode default ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt --fsname vol1 --> nfs mount mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph is my recipe correct? the cluster is set up by 3 mon/mgr nodes and 3 osd/nfs nodes, on the latters I installed one 3tb ssd, for the data, and one 300gb ssd for the journaling but my problems are : - Although I can mount the export I can't write on it - I can't understand how to use the sdc disks for journaling - I can't understand the concept of "pseudo path" here below you can find the json output of the exports --> check ceph nfs export ls nfs-cephfs ceph nfs export info nfs-cephfs /mnt ------------------------------------ json file --------- { "export_id": 1, "path": "/", "cluster_id": "nfs-cephfs", "pseudo": "/mnt", "access_type": "RW", "squash": "none", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "nfs.nfs-cephfs.1", "fs_name": "vol1" }, "clients": [] } ------------------------------------ Thanks in advance Rob
- Although I can mount the export I can't write on it
What error are you getting trying to do the write? The way you set things
up doesn't look to different than one of our integration tests for ingress over nfs ( https://github.com/ceph/ceph/blob/main/qa/suites/orch/cephadm/smoke-roleless...) and that test tests a simple read/write to the export after creating/mounting it.
- I can't understand how to use the sdc disks for journaling
you should be able to specify a `journal_devices` section in an OSD spec. For example *service_type: osdservice_id: fooplacement: hosts: - vm-00spec: data_devices: paths: - /dev/vdb journal_devices: paths: - /dev/vdc* that will make non-colocated OSDs where the devices from the journal_devices section are used as journal devices for the OSDs on the devices in the data_devices section. Although I'd recommend looking through https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s... <https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications> and see if there are any other filtering options than the path that can be used first. It's possible the path the device gets can change on reboot and you could end up with cepadm using a device you don't want it to for this as that other device gets the path another device held previously. - I can't understand the concept of "pseudo path"
I don't know at a low level either, but it seems to just be the path nfs-ganesha will present to the user. There is another argument to `ceph nfs export create` which is just "path" rather than pseudo-path that marks what actual path within the cephfs the export is mounted on. It's optional and defaults to "/" (so the export you made is mounted at the root of the fs). I think that's the one that really matters. The pseudo-path seems to just act like a user facing name for the path. On Wed, Apr 24, 2024 at 3:40 AM Roberto Maggi @ Debian <debian108@gmail.com> wrote:
Hi you all,
I'm almost new to ceph and I'm understanding, day by day, why the official support is so expansive :)
I setting up a ceph nfs network cluster whose recipe can be found here below.
#######################
--> cluster creation cephadm bootstrap --mon-ip 10.20.20.81 --cluster-network 10.20.20.0/24 --fsid $FSID --initial-dashboard-user adm \ --initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate --allow-fqdn-hostname --ssl-dashboard-port 443 \ --dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt --dashboard-key /etc/ssl/wildcard.it/wildcard.it.key \ --allow-overwrite --cleanup-on-failure cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring cephadm add-repo --release reef && cephadm install ceph-common --> adding hosts and set labels for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do ssh-copy-id -f -i /etc/ceph/ceph.pub root@$IP ; done ceph orch host add cephstage01 10.20.20.81 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage02 10.20.20.82 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage03 10.20.20.83 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstagedatanode01 10.20.20.84 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode02 10.20.20.85 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode03 10.20.20.86 --labels osd,nfs,prometheus --> network setup and daemons deploy ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24 ceph orch apply mon
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply mgr
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply prometheus
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply grafana
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply node-exporter ceph orch apply alertmanager ceph config set mgr mgr/cephadm/secure_monitoring_stack true --> disks and osd setup for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; do ssh root@$IP "hostname && wipefs -a -f /dev/sdb&& wipefs -a -f /dev/sdc"; done ceph config set mgr mgr/cephadm/device_enhanced_scan true for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device ls --hostname=$IP --wide --refresh ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdb; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdc ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdb ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdc ; done --> ganesha nfs cluster ceph mgr module enable nfs ceph fs volume create vol1 ceph nfs cluster create nfs-cephfs "cephstagedatanode01,cephstagedatanode02,cephstagedatanode03" --ingress --virtual-ip 192.168.7.80 --ingress-mode default ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt --fsname vol1 --> nfs mount mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph
is my recipe correct?
the cluster is set up by 3 mon/mgr nodes and 3 osd/nfs nodes, on the latters I installed one 3tb ssd, for the data, and one 300gb ssd for the journaling but
my problems are :
- Although I can mount the export I can't write on it
- I can't understand how to use the sdc disks for journaling
- I can't understand the concept of "pseudo path"
here below you can find the json output of the exports
--> check ceph nfs export ls nfs-cephfs ceph nfs export info nfs-cephfs /mnt ------------------------------------ json file --------- { "export_id": 1, "path": "/", "cluster_id": "nfs-cephfs", "pseudo": "/mnt", "access_type": "RW", "squash": "none", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "nfs.nfs-cephfs.1", "fs_name": "vol1" }, "clients": [] } ------------------------------------
Thanks in advance
Rob
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
first of all thanks to all! Al supposed by Robert Sander, I get "permission denied" but even writing with root privileges I get the same error. As soon as I can I'll test your suggestions and update the thread. Thanks again On 4/24/24 16:05, Adam King wrote:
- Although I can mount the export I can't write on it
What error are you getting trying to do the write? The way you set things up doesn't look to different than one of our integration tests for ingress over nfs (https://github.com/ceph/ceph/blob/main/qa/suites/orch/cephadm/smoke-roleless...) and that test tests a simple read/write to the export after creating/mounting it.
- I can't understand how to use the sdc disks for journaling
you should be able to specify a `journal_devices` section in an OSD spec. For example
*service_type: osd service_id: foo placement: hosts: - vm-00 spec: data_devices: paths: - /dev/vdb journal_devices: paths: - /dev/vdc* that will make non-colocated OSDs where the devices from the journal_devices section are used as journal devices for the OSDs on the devices in the data_devices section. Although I'd recommend looking through https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s... <https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications> and see if there are any other filtering options than the path that can be used first. It's possible the path the device gets can change on reboot and you could end up with cepadm using a device you don't want it to for this as that other device gets the path another device held previously.
- I can't understand the concept of "pseudo path"
I don't know at a low level either, but it seems to just be the path nfs-ganesha will present to the user. There is another argument to `ceph nfs export create` which is just "path" rather than pseudo-path that marks what actual path within the cephfs the export is mounted on. It's optional and defaults to "/" (so the export you made is mounted at the root of the fs). I think that's the one that really matters. The pseudo-path seems to just act like a user facing name for the path.
On Wed, Apr 24, 2024 at 3:40 AM Roberto Maggi @ Debian <debian108@gmail.com> wrote:
Hi you all,
I'm almost new to ceph and I'm understanding, day by day, why the official support is so expansive :)
I setting up a ceph nfs network cluster whose recipe can be found here below.
#######################
--> cluster creation cephadm bootstrap --mon-ip 10.20.20.81 --cluster-network 10.20.20.0/24 <http://10.20.20.0/24> --fsid $FSID --initial-dashboard-user adm \ --initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate --allow-fqdn-hostname --ssl-dashboard-port 443 \ --dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt <http://wildcard.it/wildcard.it.crt> --dashboard-key /etc/ssl/wildcard.it/wildcard.it.key <http://wildcard.it/wildcard.it.key> \ --allow-overwrite --cleanup-on-failure cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring cephadm add-repo --release reef && cephadm install ceph-common --> adding hosts and set labels for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do ssh-copy-id -f -i /etc/ceph/ceph.pub root@$IP ; done ceph orch host add cephstage01 10.20.20.81 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage02 10.20.20.82 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage03 10.20.20.83 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstagedatanode01 10.20.20.84 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode02 10.20.20.85 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode03 10.20.20.86 --labels osd,nfs,prometheus --> network setup and daemons deploy ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24 <http://10.20.20.0/24,192.168.7.0/24> ceph orch apply mon --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply mgr --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply prometheus --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply grafana --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply node-exporter ceph orch apply alertmanager ceph config set mgr mgr/cephadm/secure_monitoring_stack true --> disks and osd setup for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; do ssh root@$IP "hostname && wipefs -a -f /dev/sdb&& wipefs -a -f /dev/sdc"; done ceph config set mgr mgr/cephadm/device_enhanced_scan true for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device ls --hostname=$IP --wide --refresh ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdb; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdc ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdb ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdc ; done --> ganesha nfs cluster ceph mgr module enable nfs ceph fs volume create vol1 ceph nfs cluster create nfs-cephfs "cephstagedatanode01,cephstagedatanode02,cephstagedatanode03" --ingress --virtual-ip 192.168.7.80 --ingress-mode default ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt --fsname vol1 --> nfs mount mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph
is my recipe correct?
the cluster is set up by 3 mon/mgr nodes and 3 osd/nfs nodes, on the latters I installed one 3tb ssd, for the data, and one 300gb ssd for the journaling but
my problems are :
- Although I can mount the export I can't write on it
- I can't understand how to use the sdc disks for journaling
- I can't understand the concept of "pseudo path"
here below you can find the json output of the exports
--> check ceph nfs export ls nfs-cephfs ceph nfs export info nfs-cephfs /mnt ------------------------------------ json file --------- { "export_id": 1, "path": "/", "cluster_id": "nfs-cephfs", "pseudo": "/mnt", "access_type": "RW", "squash": "none", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "nfs.nfs-cephfs.1", "fs_name": "vol1" }, "clients": [] } ------------------------------------
Thanks in advance
Rob
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I'm not using Ceph Ganesha but GPFS Ganesha, so YMMV
ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt --fsname vol1 --> nfs mount mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph
- Although I can mount the export I can't write on it
You created a export, But there are no Clients setup https://docs.ceph.com/en/latest/mgr/nfs/#create-cephfs-export [--client_addr <value>...] It should be fine to set this to a ip range like/24 /8 or whatever you should endup with a client config like this https://github.com/nfs-ganesha/nfs-ganesha/blob/next/src/config_samples/expo...
- I can't understand the concept of "pseudo path"
Lets say your mount your exported fs to /mnt your pseudo path is /mnt, When you now mount on the machine you will end up with a folder /mnt/mnt We use this to seperate diffrent producs in gpfs. So we have a pseudo path set to /products/something, and on the machines we have the NFS mounted under /corp. the combination of both comes then to /corp/products/something On Thu, 25 Apr 2024 at 16:55, Roberto Maggi @ Debian <debian108@gmail.com> wrote:
first of all thanks to all!
Al supposed by Robert Sander, I get "permission denied" but even writing with root privileges I get the same error.
As soon as I can I'll test your suggestions and update the thread.
Thanks again
On 4/24/24 16:05, Adam King wrote:
- Although I can mount the export I can't write on it
What error are you getting trying to do the write? The way you set things up doesn't look to different than one of our integration tests for ingress over nfs (
https://github.com/ceph/ceph/blob/main/qa/suites/orch/cephadm/smoke-roleless...)
and that test tests a simple read/write to the export after creating/mounting it.
- I can't understand how to use the sdc disks for journaling
you should be able to specify a `journal_devices` section in an OSD spec. For example
*service_type: osd service_id: foo placement: hosts: - vm-00 spec: data_devices: paths: - /dev/vdb journal_devices: paths: - /dev/vdc* that will make non-colocated OSDs where the devices from the journal_devices section are used as journal devices for the OSDs on the devices in the data_devices section. Although I'd recommend looking through
https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s...
< https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications> and
see if there are any other filtering options than the path that can be used first. It's possible the path the device gets can change on reboot and you could end up with cepadm using a device you don't want it to for this as that other device gets the path another device held previously.
- I can't understand the concept of "pseudo path"
I don't know at a low level either, but it seems to just be the path nfs-ganesha will present to the user. There is another argument to `ceph nfs export create` which is just "path" rather than pseudo-path that marks what actual path within the cephfs the export is mounted on. It's optional and defaults to "/" (so the export you made is mounted at the root of the fs). I think that's the one that really matters. The pseudo-path seems to just act like a user facing name for the path.
On Wed, Apr 24, 2024 at 3:40 AM Roberto Maggi @ Debian <debian108@gmail.com> wrote:
Hi you all,
I'm almost new to ceph and I'm understanding, day by day, why the official support is so expansive :)
I setting up a ceph nfs network cluster whose recipe can be found here below.
#######################
--> cluster creation cephadm bootstrap --mon-ip 10.20.20.81 --cluster-network 10.20.20.0/24 <http://10.20.20.0/24> --fsid $FSID --initial-dashboard-user adm \ --initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate --allow-fqdn-hostname --ssl-dashboard-port 443 \ --dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt <http://wildcard.it/wildcard.it.crt> --dashboard-key /etc/ssl/wildcard.it/wildcard.it.key <http://wildcard.it/wildcard.it.key> \ --allow-overwrite --cleanup-on-failure cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring cephadm add-repo --release reef && cephadm install ceph-common --> adding hosts and set labels for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do ssh-copy-id -f -i /etc/ceph/ceph.pub root@$IP ; done ceph orch host add cephstage01 10.20.20.81 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage02 10.20.20.82 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage03 10.20.20.83 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstagedatanode01 10.20.20.84 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode02 10.20.20.85 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode03 10.20.20.86 --labels osd,nfs,prometheus --> network setup and daemons deploy ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24 <http://10.20.20.0/24,192.168.7.0/24> ceph orch apply mon
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83"
ceph orch apply mgr
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83"
ceph orch apply prometheus
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86"
ceph orch apply grafana
--placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86"
ceph orch apply node-exporter ceph orch apply alertmanager ceph config set mgr mgr/cephadm/secure_monitoring_stack true --> disks and osd setup for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; do ssh root@$IP "hostname && wipefs -a -f /dev/sdb&& wipefs -a -f /dev/sdc"; done ceph config set mgr mgr/cephadm/device_enhanced_scan true for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device ls --hostname=$IP --wide --refresh ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdb; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdc ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdb ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdc ; done --> ganesha nfs cluster ceph mgr module enable nfs ceph fs volume create vol1 ceph nfs cluster create nfs-cephfs "cephstagedatanode01,cephstagedatanode02,cephstagedatanode03" --ingress --virtual-ip 192.168.7.80 --ingress-mode default ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt --fsname vol1 --> nfs mount mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph
is my recipe correct?
the cluster is set up by 3 mon/mgr nodes and 3 osd/nfs nodes, on the latters I installed one 3tb ssd, for the data, and one 300gb ssd for the journaling but
my problems are :
- Although I can mount the export I can't write on it
- I can't understand how to use the sdc disks for journaling
- I can't understand the concept of "pseudo path"
here below you can find the json output of the exports
--> check ceph nfs export ls nfs-cephfs ceph nfs export info nfs-cephfs /mnt ------------------------------------ json file --------- { "export_id": 1, "path": "/", "cluster_id": "nfs-cephfs", "pseudo": "/mnt", "access_type": "RW", "squash": "none", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "nfs.nfs-cephfs.1", "fs_name": "vol1" }, "clients": [] } ------------------------------------
Thanks in advance
Rob
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi you all, I did the changes suggested but the situation is still the same, I set the squash to "all", since I want only "nobody:nogroup" ids but I can't understand where the path should point. If I understood it well, I pass a far disks ( unpartitioined and so unformatted ) to the osd daemon, and then I create nfs daemons, ceph will autonomously link the nfs shares on the fs managed by the osds, is it correct? Don't I have to create any filesystem on the osd? By the way this is the dump root@cephstage01:~# ceph nfs export info nfs-cephfs /mnt { "access_type": "RW", "clients": [], "cluster_id": "nfs-cephfs", "export_id": 1, "fsal": { "fs_name": "vol1", "name": "CEPH", "user_id": "nfs.nfs-cephfs.1" }, "path": "/", "protocols": [ 4 ], "pseudo": "/mnt", "security_label": true, "squash": "all", "transports": [ "TCP" ] } I can mount it correctly, but when I try to write or touch any file in it, it returns me "Permission denied" ❯ sudo mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph ❯ touch /mnt/ceph/pino touch: cannot touch '/mnt/ceph/pino': Permission denied any suggestion will be appreciated Rob On 4/24/24 16:05, Adam King wrote:
- Although I can mount the export I can't write on it
What error are you getting trying to do the write? The way you set things up doesn't look to different than one of our integration tests for ingress over nfs (https://github.com/ceph/ceph/blob/main/qa/suites/orch/cephadm/smoke-roleless...) and that test tests a simple read/write to the export after creating/mounting it.
- I can't understand how to use the sdc disks for journaling
you should be able to specify a `journal_devices` section in an OSD spec. For example
*service_type: osd service_id: foo placement: hosts: - vm-00 spec: data_devices: paths: - /dev/vdb journal_devices: paths: - /dev/vdc* that will make non-colocated OSDs where the devices from the journal_devices section are used as journal devices for the OSDs on the devices in the data_devices section. Although I'd recommend looking through https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-s... <https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications> and see if there are any other filtering options than the path that can be used first. It's possible the path the device gets can change on reboot and you could end up with cepadm using a device you don't want it to for this as that other device gets the path another device held previously.
- I can't understand the concept of "pseudo path"
I don't know at a low level either, but it seems to just be the path nfs-ganesha will present to the user. There is another argument to `ceph nfs export create` which is just "path" rather than pseudo-path that marks what actual path within the cephfs the export is mounted on. It's optional and defaults to "/" (so the export you made is mounted at the root of the fs). I think that's the one that really matters. The pseudo-path seems to just act like a user facing name for the path.
On Wed, Apr 24, 2024 at 3:40 AM Roberto Maggi @ Debian <debian108@gmail.com> wrote:
Hi you all,
I'm almost new to ceph and I'm understanding, day by day, why the official support is so expansive :)
I setting up a ceph nfs network cluster whose recipe can be found here below.
#######################
--> cluster creation cephadm bootstrap --mon-ip 10.20.20.81 --cluster-network 10.20.20.0/24 <http://10.20.20.0/24> --fsid $FSID --initial-dashboard-user adm \ --initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate --allow-fqdn-hostname --ssl-dashboard-port 443 \ --dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt <http://wildcard.it/wildcard.it.crt> --dashboard-key /etc/ssl/wildcard.it/wildcard.it.key <http://wildcard.it/wildcard.it.key> \ --allow-overwrite --cleanup-on-failure cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring cephadm add-repo --release reef && cephadm install ceph-common --> adding hosts and set labels for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do ssh-copy-id -f -i /etc/ceph/ceph.pub root@$IP ; done ceph orch host add cephstage01 10.20.20.81 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage02 10.20.20.82 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage03 10.20.20.83 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstagedatanode01 10.20.20.84 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode02 10.20.20.85 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode03 10.20.20.86 --labels osd,nfs,prometheus --> network setup and daemons deploy ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24 <http://10.20.20.0/24,192.168.7.0/24> ceph orch apply mon --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply mgr --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply prometheus --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply grafana --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply node-exporter ceph orch apply alertmanager ceph config set mgr mgr/cephadm/secure_monitoring_stack true --> disks and osd setup for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; do ssh root@$IP "hostname && wipefs -a -f /dev/sdb&& wipefs -a -f /dev/sdc"; done ceph config set mgr mgr/cephadm/device_enhanced_scan true for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device ls --hostname=$IP --wide --refresh ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdb; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdc ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdb ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdc ; done --> ganesha nfs cluster ceph mgr module enable nfs ceph fs volume create vol1 ceph nfs cluster create nfs-cephfs "cephstagedatanode01,cephstagedatanode02,cephstagedatanode03" --ingress --virtual-ip 192.168.7.80 --ingress-mode default ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt --fsname vol1 --> nfs mount mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph
is my recipe correct?
the cluster is set up by 3 mon/mgr nodes and 3 osd/nfs nodes, on the latters I installed one 3tb ssd, for the data, and one 300gb ssd for the journaling but
my problems are :
- Although I can mount the export I can't write on it
- I can't understand how to use the sdc disks for journaling
- I can't understand the concept of "pseudo path"
here below you can find the json output of the exports
--> check ceph nfs export ls nfs-cephfs ceph nfs export info nfs-cephfs /mnt ------------------------------------ json file --------- { "export_id": 1, "path": "/", "cluster_id": "nfs-cephfs", "pseudo": "/mnt", "access_type": "RW", "squash": "none", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "nfs.nfs-cephfs.1", "fs_name": "vol1" }, "clients": [] } ------------------------------------
Thanks in advance
Rob
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 4/29/24 17:21, Roberto Maggi @ Debian wrote:
I can mount it correctly, but when I try to write or touch any file in it, it returns me "Permission denied"
Works as expected. You have set squash to all, now every NFS client is mapped to nobody on the server. But only root is able to write to the CephFS at first. Set squash to "no_root_squash" to be able to write as root to the NFS share. Create a directory and change its permissions to someone else. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Wow, you made it farther than I did. I got it installed, added hosts, then NOTHING. It showed there were physical disks on the hosts but wouldn't create the OSDs. Command was accepted, but NOTHING happened. No output, no error, no NOTHING. I fought with it for over a week and finally gave up, as with no feedback as to what the issue is it's impossible to troubleshoot. A product that does NOTHING isn't a product at all. I posted a detailed message here with screenshots, steps, everything somebody would need to reproduce my situation. The post got blocked because it was too big and sent for moderation. It never got approved or rejected. So I moved on, can't be using something that does NOTHING with no way to proceed past that point. ________________________________ From: Roberto Maggi @ Debian <debian108@gmail.com> Sent: April 24, 2024 01:39 To: Ceph Users <ceph-users@ceph.io> Subject: [ceph-users] ceph recipe for nfs exports [You don't often get email from debian108@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Hi you all, I'm almost new to ceph and I'm understanding, day by day, why the official support is so expansive :) I setting up a ceph nfs network cluster whose recipe can be found here below. ####################### --> cluster creation cephadm bootstrap --mon-ip 10.20.20.81 --cluster-network 10.20.20.0/24 --fsid $FSID --initial-dashboard-user adm \ --initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate --allow-fqdn-hostname --ssl-dashboard-port 443 \ --dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt --dashboard-key /etc/ssl/wildcard.it/wildcard.it.key \ --allow-overwrite --cleanup-on-failure cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring cephadm add-repo --release reef && cephadm install ceph-common --> adding hosts and set labels for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do ssh-copy-id -f -i /etc/ceph/ceph.pub root@$IP ; done ceph orch host add cephstage01 10.20.20.81 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage02 10.20.20.82 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage03 10.20.20.83 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstagedatanode01 10.20.20.84 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode02 10.20.20.85 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode03 10.20.20.86 --labels osd,nfs,prometheus --> network setup and daemons deploy ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24 ceph orch apply mon --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply mgr --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply prometheus --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply grafana --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply node-exporter ceph orch apply alertmanager ceph config set mgr mgr/cephadm/secure_monitoring_stack true --> disks and osd setup for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; do ssh root@$IP "hostname && wipefs -a -f /dev/sdb&& wipefs -a -f /dev/sdc"; done ceph config set mgr mgr/cephadm/device_enhanced_scan true for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device ls --hostname=$IP --wide --refresh ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdb; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdc ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdb ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdc ; done --> ganesha nfs cluster ceph mgr module enable nfs ceph fs volume create vol1 ceph nfs cluster create nfs-cephfs "cephstagedatanode01,cephstagedatanode02,cephstagedatanode03" --ingress --virtual-ip 192.168.7.80 --ingress-mode default ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt --fsname vol1 --> nfs mount mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph is my recipe correct? the cluster is set up by 3 mon/mgr nodes and 3 osd/nfs nodes, on the latters I installed one 3tb ssd, for the data, and one 300gb ssd for the journaling but my problems are : - Although I can mount the export I can't write on it - I can't understand how to use the sdc disks for journaling - I can't understand the concept of "pseudo path" here below you can find the json output of the exports --> check ceph nfs export ls nfs-cephfs ceph nfs export info nfs-cephfs /mnt ------------------------------------ json file --------- { "export_id": 1, "path": "/", "cluster_id": "nfs-cephfs", "pseudo": "/mnt", "access_type": "RW", "squash": "none", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "nfs.nfs-cephfs.1", "fs_name": "vol1" }, "clients": [] } ------------------------------------ Thanks in advance Rob _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 4/24/24 09:39, Roberto Maggi @ Debian wrote:
ceph orch host add cephstage01 10.20.20.81 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage02 10.20.20.82 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage03 10.20.20.83 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstagedatanode01 10.20.20.84 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode02 10.20.20.85 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode03 10.20.20.86 --labels osd,nfs,prometheus --> network setup and daemons deploy ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24 ceph orch apply mon --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply mgr --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply prometheus --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply grafana --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86"
Two remarks here: - You are labeling all the hosts but then you use a hostname based placement strategy for the services. Why not use the labels for placing the services? - Usually you only need one Prometheus, one Grafana and one Alert-Manager in the cluster. There is no need to deploy these on each host.
ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt --fsname vol1 --> nfs mount mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph
is my recipe correct?
Apart from the remarks above it should get you a working NFS export.
- Although I can mount the export I can't write on it
You have not specified a value for --squash when creating the export. Your CephFS is empty and the root directory is only writable by the root user, but this gets "squashed" to nobody when using NFS.
- I can't understand how to use the sdc disks for journaling
When all your devices are SSD you do not need "journaling" (which is today the RocksDB and WAL).
- I can't understand the concept of "pseudo path"
This is an NFSv4 concept. It allows to mount a virtual root of the NFS server and access all exports below it without having to mount each one separately. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
participants (5)
-
Adam King
-
Ceph.io
-
P Wagner-Beccard
-
Robert Sander
-
Roberto Maggi ＠ Debian