Error EINVAL: check-host failed - Failed to add host
Hello, Please help: CEPH cluster using docker. using below command for the bootstrap with provided key and pub cephadm -v bootstrap --mon-ip <ip> --allow-overwrite --ssh-private-key id_ed25519 --ssh-public-key id_ed25519.pub able to ssh directly with id_ed25519 key. RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=<hostname> -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/85c044a8-1d82-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp1sw6a5s0:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpethxlwxr:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch host add <hostname> <ip>: Error EINVAL: check-host failed: Unable to write <host>:/var/lib/ceph/85c044a8-1d82-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed also validated below and it was successful:
ceph cephadm get-ssh-config > ssh_config ceph config-key get mgr/cephadm/ssh_identity_key > key ssh -F ssh_config -i key root@<hostname>
--- root@<hostname>:~/.ssh# ceph orch host add <hostname> <ip> Error EINVAL: check-host failed: Unable to write <hostname>:/var/lib/ceph/fe8ecd30-1da2-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed root@<hostname>:~/.ssh# Thanks, Surya
Hi, I think there's something else wrong with your setup, I could bootstrap a cluster without an issue with ed keys: ceph:~ # ssh-keygen -t ed25519 Generating public/private ed25519 key pair. ceph:~ # cephadm --image quay.io/ceph/ceph:v18.2.2 bootstrap --mon-ip [IP] [some more options] --ssh-private-key .ssh/id_ed25519 --ssh-public-key .ssh/id_ed25519.pub ... Using provided ssh keys... Adding key to root@localhost authorized_keys... Adding host ceph... ... Bootstrap complete. ceph:~ # ceph cephadm get-pub-key ssh-ed25519 AAAAC... root@ceph Regards, Eugen Zitat von isnraju26@yahoo.com:
Hello,
Please help: CEPH cluster using docker.
using below command for the bootstrap with provided key and pub
cephadm -v bootstrap --mon-ip <ip> --allow-overwrite --ssh-private-key id_ed25519 --ssh-public-key id_ed25519.pub
able to ssh directly with id_ed25519 key.
RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=<hostname> -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/85c044a8-1d82-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp1sw6a5s0:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpethxlwxr:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch host add <hostname> <ip>: Error EINVAL: check-host failed: Unable to write <host>:/var/lib/ceph/85c044a8-1d82-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed
also validated below and it was successful:
ceph cephadm get-ssh-config > ssh_config ceph config-key get mgr/cephadm/ssh_identity_key > key ssh -F ssh_config -i key root@<hostname>
--- root@<hostname>:~/.ssh# ceph orch host add <hostname> <ip> Error EINVAL: check-host failed: Unable to write <hostname>:/var/lib/ceph/fe8ecd30-1da2-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed root@<hostname>:~/.ssh#
Thanks, Surya _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the reply @Eugen Block Yes, there is some thing else is wrong in my server, but no clue why it's failing and what is the cause of boortstrap failure. I was able to bootstrap with ed keys in other server.
It sounded like you were pointing towards the ssh key as a possible root cause, you didn't mention that it worked on other clusters. Then you'll need to compare the settings between a working host and the failing one, check the ceohadm.log for more details. You could also execute the check-host command manually, with --verbose flag: cephadm --verbose check-host --expect-hostname <hostname> cephadm --verbose prepare-host Does that show anything useful? Zitat von isnraju26@yahoo.com:
Thanks for the reply @Eugen Block
Yes, there is some thing else is wrong in my server, but no clue why it's failing and what is the cause of boortstrap failure.
I was able to bootstrap with ed keys in other server. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes looks it's issue related to SSH, but not sure where is the problem to fix this. ---- root@cephhost01:~# cephadm --verbose check-host --expect-hostname cephhost01 -------------------------------------------------------------------------------- cephadm ['--verbose', 'check-host', '--expect-hostname', 'cephhost01'] docker (/usr/bin/docker) is present systemctl is present lvcreate is present systemctl: stdout enabled systemctl: stdout active Unit chrony.service is enabled and running Hostname "cephhost01" matches what is expected. Host looks OK root@cephhost01:~# cephadm --verbose prepare-host -------------------------------------------------------------------------------- cephadm ['--verbose', 'prepare-host'] Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... systemctl: stdout enabled systemctl: stdout active Unit chrony.service is enabled and running Repeating the final host check... docker (/usr/bin/docker) is present systemctl is present lvcreate is present systemctl: stdout enabled systemctl: stdout active Unit chrony.service is enabled and running Host looks OK root@cephhost01:~# root@cephhost01:~# ceph orch host add cephhost01 10.11.11.101 Error EINVAL: check-host failed: Unable to write cephhost01:/var/lib/ceph/4c2b76e0-230e-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a25 8bde893227c262c30bca8d: Session request failed root@cephhost01:~# have python3 and OS versions are same as worked server. bootstrap error with debug. Traceback (most recent call last): File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 5808, in prepare_ssh File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 6317, in cli File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 4673, in run File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 2135, in call_throws RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=cephhost01 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ce ph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp96ev82_w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp 2hnnuu16:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch host add mon 10.11.11.101: Error EINVAL: check-host failed: Unable to write cephhost01:/var/lib/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a25 8bde893227c262c30bca8d: Session request failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 10700, in <module> File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 10688, in main File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 6156, in _rollback File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 2495, in _default_image File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 6382, in command_bootstrap File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 5810, in prepare_ssh __main__.Error: Failed to add host <cephhost01>: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --uli mit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=cephhost01 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp96ev82_w:/etc/ceph/ce ph.client.admin.keyring:z -v /tmp/ceph-tmp2hnnuu16:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch host add cephhost01 10.11.11.101: Error EINVAL: check-host failed: Unable to write cephhost01:/var/lib/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a25 8bde893227c262c30bca8d: Session request failed ceph.log 2024-06-05 11:36:59,726 7f2392a5f780 INFO Using provided ssh keys... 2024-06-05 11:37:01,885 7f2392a5f780 DEBUG /usr/bin/ceph: stdout ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICbhHKV4COtHgmUSClF0tdfcx4WF5OeOBt IvTLj8eyzA root@cephhost01 2024-06-05 11:37:01,885 7f2392a5f780 INFO Adding key to root@localhost authorized_keys... 2024-06-05 11:37:01,886 7f2392a5f780 INFO key already in root@localhost authorized_keys... 2024-06-05 11:37:01,886 7f2392a5f780 INFO Adding host cephhost01... 2024-06-05 11:37:03,759 7f2392a5f780 INFO Non-zero exit code 22 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=cephhost01 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp96ev82_w:/etc/ceph/ceph. client.admin.keyring:z -v /tmp/ceph-tmp2hnnuu16:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch host add cephhost01 10.11.11.101 2024-06-05 11:37:03,759 7f2392a5f780 INFO /usr/bin/ceph: stderr Error EINVAL: check-host failed: 2024-06-05 11:37:03,759 7f2392a5f780 INFO /usr/bin/ceph: stderr Unable to write cephhost01:/var/lib/ceph/d5d1b7c6-232f-11ef-9ea1 -a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed 2024-06-05 11:37:03,759 7f2392a5f780 ERROR Error: Failed to add host <cephhost01>: Failed command: /usr/bin/docker run --rm --ip c=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph: v18 -e NODE_NAME=cephhost01 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp96ev82_w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2hnnuu16:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch h ost add cephhost01 10.11.11.101: Error EINVAL: check-host failed: Unable to write cephhost01:/var/lib/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a25 8bde893227c262c30bca8d: Session request failed Able to SSH with ED key. root@cephhost01:~/.ssh# ll total 28 drwx------ 2 root root 111 May 23 09:47 ./ drwx------ 10 root root 4096 Jun 5 11:44 ../ -rw------- 1 root root 104 May 23 09:49 authorized_keys -rw------- 1 root root 419 May 23 09:38 id_ed25519 -rw-r----- 1 root root 104 May 23 09:38 id_ed25519.pub -rw------- 1 root root 898 May 23 12:22 known_hosts -rw-r--r-- 1 root root 142 May 23 09:46 known_hosts.old root@cephhost01:~/.ssh# ssh -i id_ed25519 cephhost01 Authorized uses only. All activity may be monitored and reported. Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-94-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/pro You have mail. Last login: Wed May 29 11:19:19 2024 from 10.11.11.101 root@cephhost01:~# ------
Can you paste the output of: ls -l /var/lib/ceph/ on cephhost01? It says it can't write to that directory: Unable to write cephhost01:/var/lib/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a25 Which distro are you using? Zitat von isnraju26@yahoo.com:
Yes looks it's issue related to SSH, but not sure where is the problem to fix this.
---- root@cephhost01:~# cephadm --verbose check-host --expect-hostname cephhost01 -------------------------------------------------------------------------------- cephadm ['--verbose', 'check-host', '--expect-hostname', 'cephhost01'] docker (/usr/bin/docker) is present systemctl is present lvcreate is present systemctl: stdout enabled systemctl: stdout active Unit chrony.service is enabled and running Hostname "cephhost01" matches what is expected. Host looks OK root@cephhost01:~# cephadm --verbose prepare-host -------------------------------------------------------------------------------- cephadm ['--verbose', 'prepare-host'] Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... systemctl: stdout enabled systemctl: stdout active Unit chrony.service is enabled and running Repeating the final host check... docker (/usr/bin/docker) is present systemctl is present lvcreate is present systemctl: stdout enabled systemctl: stdout active Unit chrony.service is enabled and running Host looks OK root@cephhost01:~#
root@cephhost01:~# ceph orch host add cephhost01 10.11.11.101 Error EINVAL: check-host failed: Unable to write cephhost01:/var/lib/ceph/4c2b76e0-230e-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a25 8bde893227c262c30bca8d: Session request failed root@cephhost01:~#
have python3 and OS versions are same as worked server.
bootstrap error with debug.
Traceback (most recent call last): File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 5808, in prepare_ssh File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 6317, in cli File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 4673, in run File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 2135, in call_throws RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=cephhost01 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ce ph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp96ev82_w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp 2hnnuu16:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch host add mon 10.11.11.101: Error EINVAL: check-host failed: Unable to write cephhost01:/var/lib/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a25 8bde893227c262c30bca8d: Session request failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 10700, in <module> File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 10688, in main File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 6156, in _rollback File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 2495, in _default_image File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 6382, in command_bootstrap File "/tmp/tmpwf3vvwn_.cephadm.build/__main__.py", line 5810, in prepare_ssh __main__.Error: Failed to add host <cephhost01>: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --uli mit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=cephhost01 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp96ev82_w:/etc/ceph/ce ph.client.admin.keyring:z -v /tmp/ceph-tmp2hnnuu16:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch host add cephhost01 10.11.11.101: Error EINVAL: check-host failed: Unable to write cephhost01:/var/lib/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a25 8bde893227c262c30bca8d: Session request failed
ceph.log
2024-06-05 11:36:59,726 7f2392a5f780 INFO Using provided ssh keys... 2024-06-05 11:37:01,885 7f2392a5f780 DEBUG /usr/bin/ceph: stdout ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICbhHKV4COtHgmUSClF0tdfcx4WF5OeOBt IvTLj8eyzA root@cephhost01 2024-06-05 11:37:01,885 7f2392a5f780 INFO Adding key to root@localhost authorized_keys... 2024-06-05 11:37:01,886 7f2392a5f780 INFO key already in root@localhost authorized_keys... 2024-06-05 11:37:01,886 7f2392a5f780 INFO Adding host cephhost01... 2024-06-05 11:37:03,759 7f2392a5f780 INFO Non-zero exit code 22 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=cephhost01 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp96ev82_w:/etc/ceph/ceph. client.admin.keyring:z -v /tmp/ceph-tmp2hnnuu16:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch host add cephhost01 10.11.11.101 2024-06-05 11:37:03,759 7f2392a5f780 INFO /usr/bin/ceph: stderr Error EINVAL: check-host failed: 2024-06-05 11:37:03,759 7f2392a5f780 INFO /usr/bin/ceph: stderr Unable to write cephhost01:/var/lib/ceph/d5d1b7c6-232f-11ef-9ea1 -a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed 2024-06-05 11:37:03,759 7f2392a5f780 ERROR Error: Failed to add host <cephhost01>: Failed command: /usr/bin/docker run --rm --ip c=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph: v18 -e NODE_NAME=cephhost01 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp96ev82_w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp2hnnuu16:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch h ost add cephhost01 10.11.11.101: Error EINVAL: check-host failed: Unable to write cephhost01:/var/lib/ceph/d5d1b7c6-232f-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a25 8bde893227c262c30bca8d: Session request failed
Able to SSH with ED key.
root@cephhost01:~/.ssh# ll total 28 drwx------ 2 root root 111 May 23 09:47 ./ drwx------ 10 root root 4096 Jun 5 11:44 ../ -rw------- 1 root root 104 May 23 09:49 authorized_keys -rw------- 1 root root 419 May 23 09:38 id_ed25519 -rw-r----- 1 root root 104 May 23 09:38 id_ed25519.pub -rw------- 1 root root 898 May 23 12:22 known_hosts -rw-r--r-- 1 root root 142 May 23 09:46 known_hosts.old root@cephhost01:~/.ssh# ssh -i id_ed25519 cephhost01 Authorized uses only. All activity may be monitored and reported. Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-94-generic x86_64)
* Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/pro You have mail. Last login: Wed May 29 11:19:19 2024 from 10.11.11.101 root@cephhost01:~# ------ _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
isnraju26＠yahoo.com