Hi, I am trying to follow the documentation at https://docs.ceph.com/en/reef/rbd/nvmeof-target-configure/ to deploy an NVMe over Fabric service. Step 2b of the configuration section is currently the showstopper. First the command says: error: the following arguments are required: --host-name/-t Then it tells me (after adding --host-name): error: unrecognized arguments: --gateway-name XXX and when I remove --gateway-name the error is: both gateway_name and traddr or neither must be specified So I am stuck in a kind of a loop here. Is there a working description for NVMe over TCP available? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, On 5/30/24 11:58, Robert Sander wrote:
I am trying to follow the documentation at https://docs.ceph.com/en/reef/rbd/nvmeof-target-configure/ to deploy an NVMe over Fabric service.
It looks like the cephadm orchestrator in this 18.2.2 cluster uses the image quay.io/ceph/nvmeof:0.0.2 which is 9 months old. When I try to redeploy the daemon with the latest image ceph orch daemon redeploy nvmeof.nvme01.cephtest29.gookea --image quay.io/ceph/nvmeof:latest it tells me: Error EINVAL: Cannot redeploy nvmeof.nvme01.cephtest29.gookea with a new image: Supported types are: mgr, mon, crash, osd, mds, rgw, rbd-mirror, cephfs-mirror, ceph-exporter, iscsi, nfs How do I set the container image for this service? ceph config set nvmeof container_image quay.io/ceph/nvmeof:latest does not work with Error EINVAL: unrecognized config target 'nvmeof' Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hello Robert, You could try: ceph config set mgr mgr/cephadm/container_image_nvmeof "quay.io/ceph/nvmeof:1.2.13" or whatever image tag you need (1.2.13 is current latest). Another way to run the image is by editing the unit.run file of the service or by directly running the container with podman run (you'll need to adjust names, cluster sid, etc.): /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --init --name ceph-aa558815-042c-4fce-ac37-80c0255bf3c0-nvmeof-nvmeof_pool01-test-lis04h02-baakhx --pids-limit=-1 --ulimit memlock=-1:-1 --ulimit nofile=10240 --cap-add=SYS_ADMIN --cap-add=CAP_SYS_NICE --log-driver journald --conmon-pidfile /run/ceph-aa558815-042c-4fce-ac37-80c0255bf3c0@nvmeof.nvmeof_pool01.test-lis04h02.baakhx.service-pid --cidfile /run/ceph-aa558815-042c-4fce-ac37-80c0255bf3c0@nvmeof.nvmeof_pool01.test-lis04h02.baakhx.service-cid --cgroups=split -e CONTAINER_IMAGE=quay.io/ceph/nvmeof:1.2.13 -e NODE_NAME=test-lis04h02.peta.libe.dc.univ-lorraine.fr -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/aa558815-042c-4fce-ac37-80c0255bf3c0/nvmeof.nvmeof_pool01.test-lis04h02.baakhx/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/aa558815-042c-4fce-ac37-80c0255bf3c0/nvmeof.nvmeof_pool01.test-lis04h02.baakhx/keyring:/etc/ceph/keyring:z -v /var/lib/ceph/aa558815-042c-4fce-ac37-80c0255bf3c0/nvmeof.nvmeof_pool01.test-lis04h02.baakhx/ceph-nvmeof.conf:/src/ceph-nvmeof.conf:z -v /var/lib/ceph/aa558815-042c-4fce-ac37-80c0255bf3c0/nvmeof.nvmeof_pool01.test-lis04h02.baakhx/configfs:/sys/kernel/config -v /dev/hugepages:/dev/hugepages -v /dev/vfio/vfio:/dev/vfio/vfio -v /etc/hosts:/etc/hosts:ro --mount type=bind,source=/lib/modules,destination=/lib/modules,ro=true quay.io/ceph/nvmeof:1.2.13 The commands I wrote here [1] in February should still work I believe. Regards, Frédéric. [1] https://github.com/ceph/ceph-nvmeof/issues/459 ----- Le 30 Mai 24, à 13:03, Robert Sander r.sander@heinlein-support.de a écrit :
Hi,
On 5/30/24 11:58, Robert Sander wrote:
I am trying to follow the documentation at https://docs.ceph.com/en/reef/rbd/nvmeof-target-configure/ to deploy an NVMe over Fabric service.
It looks like the cephadm orchestrator in this 18.2.2 cluster uses the image quay.io/ceph/nvmeof:0.0.2 which is 9 months old.
When I try to redeploy the daemon with the latest image ceph orch daemon redeploy nvmeof.nvme01.cephtest29.gookea --image quay.io/ceph/nvmeof:latest it tells me:
Error EINVAL: Cannot redeploy nvmeof.nvme01.cephtest29.gookea with a new image: Supported types are: mgr, mon, crash, osd, mds, rgw, rbd-mirror, cephfs-mirror, ceph-exporter, iscsi, nfs
How do I set the container image for this service?
ceph config set nvmeof container_image quay.io/ceph/nvmeof:latest
does not work with Error EINVAL: unrecognized config target 'nvmeof'
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 5/30/24 14:18, Frédéric Nass wrote:
ceph config set mgr mgr/cephadm/container_image_nvmeof "quay.io/ceph/nvmeof:1.2.13"
Thanks for the hint. With that the orchestrator deploys the current container image. But: It suddenly listens on port 5499 instead of 5500 and: # podman run -it quay.io/ceph/nvmeof-cli:latest --server-address 10.128.8.29 --server-port 5500 subsystem add --subsystem nqn.2016-06.io.spdk:cephtest29 Failure adding subsystem nqn.2016-06.io.spdk:cephtest29: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.128.8.29:5500: Failed to connect to remote host: Connection refused" debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.128.8.29:5500: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2024-05-30T13:59:33.24226686+00:00"}" # podman run -it quay.io/ceph/nvmeof-cli:latest --server-address 10.128.8.29 --server-port 5499 subsystem add --subsystem nqn.2016-06.io.spdk:cephtest29 Failure adding subsystem nqn.2016-06.io.spdk:cephtest29: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNIMPLEMENTED details = "Method not found!" debug_error_string = "UNKNOWN:Error received from peer ipv4:10.128.8.29:5499 {created_time:"2024-05-30T13:59:49.678809906+00:00", grpc_status:12, grpc_message:"Method not found!"}" Is this not production ready? Why is it in the documentation for a released Ceph version? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
There's a major NVMe effort underway but it's not even merged to master yet, so I'm not sure how docs would have ended up in the Reef doc tree. :/ Zac, any idea? Can we pull this out? -Greg On Thu, May 30, 2024 at 7:03 AM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
On 5/30/24 14:18, Frédéric Nass wrote:
ceph config set mgr mgr/cephadm/container_image_nvmeof "quay.io/ceph/nvmeof:1.2.13"
Thanks for the hint. With that the orchestrator deploys the current container image.
But: It suddenly listens on port 5499 instead of 5500 and:
# podman run -it quay.io/ceph/nvmeof-cli:latest --server-address 10.128.8.29 --server-port 5500 subsystem add --subsystem nqn.2016-06.io.spdk:cephtest29 Failure adding subsystem nqn.2016-06.io.spdk:cephtest29: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.128.8.29:5500: Failed to connect to remote host: Connection refused" debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.128.8.29:5500: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2024-05-30T13:59:33.24226686+00:00"}"
# podman run -it quay.io/ceph/nvmeof-cli:latest --server-address 10.128.8.29 --server-port 5499 subsystem add --subsystem nqn.2016-06.io.spdk:cephtest29 Failure adding subsystem nqn.2016-06.io.spdk:cephtest29: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNIMPLEMENTED details = "Method not found!" debug_error_string = "UNKNOWN:Error received from peer ipv4:10.128.8.29:5499 {created_time:"2024-05-30T13:59:49.678809906+00:00", grpc_status:12, grpc_message:"Method not found!"}"
Is this not production ready? Why is it in the documentation for a released Ceph version?
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Thursday, May 30, 2024 7:03:44 AM EDT Robert Sander wrote:
Hi,
On 5/30/24 11:58, Robert Sander wrote:
I am trying to follow the documentation at https://docs.ceph.com/en/reef/rbd/nvmeof-target-configure/ to deploy an NVMe over Fabric service.
It looks like the cephadm orchestrator in this 18.2.2 cluster uses the image quay.io/ceph/nvmeof:0.0.2 which is 9 months old.
When I try to redeploy the daemon with the latest image ceph orch daemon redeploy nvmeof.nvme01.cephtest29.gookea --image quay.io/ceph/nvmeof:latest it tells me:
Error EINVAL: Cannot redeploy nvmeof.nvme01.cephtest29.gookea with a new image: Supported types are: mgr, mon, crash, osd, mds, rgw, rbd-mirror, cephfs-mirror, ceph-exporter, iscsi, nfs
How do I set the container image for this service?
ceph config set nvmeof container_image quay.io/ceph/nvmeof:latest
I haven't tested this but try this instead: ceph config set mgr mgr/cephadm/container_image_nvmeof quay.io/ceph/ nvmeof:latest Generally it is the cephadm component that controls what images get deployed. The pattern has been to name a config key within the cephadm mgr module "container_image_<thing>".
does not work with Error EINVAL: unrecognized config target 'nvmeof'
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I've never used this feature, but I wanted to point out your command versus the error message; gateway-name / gateway_name (dash versus underscore) On Thu, May 30, 2024 at 5:07 AM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
I am trying to follow the documentation at https://docs.ceph.com/en/reef/rbd/nvmeof-target-configure/ to deploy an NVMe over Fabric service.
Step 2b of the configuration section is currently the showstopper.
First the command says:
error: the following arguments are required: --host-name/-t
Then it tells me (after adding --host-name):
error: unrecognized arguments: --gateway-name XXX
and when I remove --gateway-name the error is:
both gateway_name and traddr or neither must be specified
So I am stuck in a kind of a loop here.
Is there a working description for NVMe over TCP available?
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ______________________________ Dino Yancey
participants (5)
-
Dino Yancey
-
Frédéric Nass
-
Gregory Farnum
-
John Mulligan
-
Robert Sander