non root deploy ceph 17.2.5 failed
Hi, with following command: sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below: debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e: scp: /tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new: Permission denied Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file await asyncssh.scp(f.name, (conn, tmp_path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp await source.run(srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run await self._send_files(path, b'') File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files await self._send_file(srcpath, dstpath, attrs) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file await self._make_cd_request(b'C', attrs, size, srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request self._fs.basename(path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc Any ideas on this? Thanks, Ben
Hi, could you provide some more details about your host OS? Which cephadm version is it? I was able to bootstrap a one-node cluster with both 17.2.5 and 17.2.6 with a non-root user with no such error on openSUSE Leap 15.4: quincy:~ # rpm -qa | grep cephadm cephadm-17.2.6.248+gad656d572cb-lp154.2.1.noarch deployer@quincy:~> sudo cephadm --image quay.io/ceph/ceph:v17.2.5 bootstrap --mon-ip 172.17.2.3 --skip-monitoring-stack --ssh-user deployer --single-host-defaults Verifying ssh connectivity ... Adding key to deployer@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... podman (/usr/bin/podman) version 4.4.4 is present [...] Ceph version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available [...] Adding key to deployer@localhost authorized_keys... Adding host quincy... Deploying mon service with default placement... Deploying mgr service with default placement... [...] Bootstrap complete. Zitat von Ben <ruidong.gao@gmail.com>:
Hi,
with following command:
sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below:
debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e: scp: /tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new: Permission denied
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file
await asyncssh.scp(f.name, (conn, tmp_path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
await source.run(srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
await self._send_files(path, b'')
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
await self._send_file(srcpath, dstpath, attrs)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
await self._make_cd_request(b'C', attrs, size, srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
self._fs.basename(path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
raise exc
Any ideas on this?
Thanks, Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, It is uos v20(with kernel 4.19), one linux distribution among others. no matter since cephadm deploys things in containers by default. cephadm is pulled by curl from Quincy branch of github. I think you could see some sort of errors if you remove parameter --single-host-defaults. More investigation shows it looks like a bug with cephadm. during the deploying procedure ,/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new is created through sudo ssh session remotely(with owner of root) and /tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/ is changed to owner of ssh user deployer. The correct thing to do instead is, /tmp/var/ be changed to the owner deployer recursively so that following scp can have access permission. I will see if having time to wire up a PR to fix it. Thanks for help on this. Ben Eugen Block <eblock@nde.ag> 于2023年5月8日周一 21:01写道：
Hi,
could you provide some more details about your host OS? Which cephadm version is it? I was able to bootstrap a one-node cluster with both 17.2.5 and 17.2.6 with a non-root user with no such error on openSUSE Leap 15.4:
quincy:~ # rpm -qa | grep cephadm cephadm-17.2.6.248+gad656d572cb-lp154.2.1.noarch
deployer@quincy:~> sudo cephadm --image quay.io/ceph/ceph:v17.2.5 bootstrap --mon-ip 172.17.2.3 --skip-monitoring-stack --ssh-user deployer --single-host-defaults Verifying ssh connectivity ... Adding key to deployer@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... podman (/usr/bin/podman) version 4.4.4 is present [...] Ceph version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available [...] Adding key to deployer@localhost authorized_keys... Adding host quincy... Deploying mon service with default placement... Deploying mgr service with default placement... [...] Bootstrap complete.
Zitat von Ben <ruidong.gao@gmail.com>:
Hi,
with following command:
sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below:
debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write
szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e:
scp:
/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new:
Permission denied
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file
await asyncssh.scp(f.name, (conn, tmp_path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
await source.run(srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
await self._send_files(path, b'')
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
await self._send_file(srcpath, dstpath, attrs)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
await self._make_cd_request(b'C', attrs, size, srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
self._fs.basename(path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
raise exc
Any ideas on this?
Thanks, Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I just retried without the single-host option and it worked. Also everything under /tmp/var belongs to root in my case. Unfortunately, I can't use the curl-based cephadm but the contents are identical, I compared. Not sure what it could be at the moment. Zitat von Ben <ruidong.gao@gmail.com>:
Hi, It is uos v20(with kernel 4.19), one linux distribution among others. no matter since cephadm deploys things in containers by default. cephadm is pulled by curl from Quincy branch of github.
I think you could see some sort of errors if you remove parameter --single-host-defaults.
More investigation shows it looks like a bug with cephadm. during the deploying procedure ,/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new is created through sudo ssh session remotely(with owner of root) and /tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/ is changed to owner of ssh user deployer. The correct thing to do instead is, /tmp/var/ be changed to the owner deployer recursively so that following scp can have access permission. I will see if having time to wire up a PR to fix it.
Thanks for help on this. Ben
Eugen Block <eblock@nde.ag> 于2023年5月8日周一 21:01写道：
Hi,
could you provide some more details about your host OS? Which cephadm version is it? I was able to bootstrap a one-node cluster with both 17.2.5 and 17.2.6 with a non-root user with no such error on openSUSE Leap 15.4:
quincy:~ # rpm -qa | grep cephadm cephadm-17.2.6.248+gad656d572cb-lp154.2.1.noarch
deployer@quincy:~> sudo cephadm --image quay.io/ceph/ceph:v17.2.5 bootstrap --mon-ip 172.17.2.3 --skip-monitoring-stack --ssh-user deployer --single-host-defaults Verifying ssh connectivity ... Adding key to deployer@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... podman (/usr/bin/podman) version 4.4.4 is present [...] Ceph version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available [...] Adding key to deployer@localhost authorized_keys... Adding host quincy... Deploying mon service with default placement... Deploying mgr service with default placement... [...] Bootstrap complete.
Zitat von Ben <ruidong.gao@gmail.com>:
Hi,
with following command:
sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below:
debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write
szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e:
scp:
/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new:
Permission denied
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file
await asyncssh.scp(f.name, (conn, tmp_path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
await source.run(srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
await self._send_files(path, b'')
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
await self._send_file(srcpath, dstpath, attrs)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
await self._make_cd_request(b'C', attrs, size, srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
self._fs.basename(path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
raise exc
Any ideas on this?
Thanks, Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
And I just tried with docker as well, works too. Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I just retried without the single-host option and it worked. Also everything under /tmp/var belongs to root in my case. Unfortunately, I can't use the curl-based cephadm but the contents are identical, I compared. Not sure what it could be at the moment.
Zitat von Ben <ruidong.gao@gmail.com>:
Hi, It is uos v20(with kernel 4.19), one linux distribution among others. no matter since cephadm deploys things in containers by default. cephadm is pulled by curl from Quincy branch of github.
I think you could see some sort of errors if you remove parameter --single-host-defaults.
More investigation shows it looks like a bug with cephadm. during the deploying procedure ,/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new is created through sudo ssh session remotely(with owner of root) and /tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/ is changed to owner of ssh user deployer. The correct thing to do instead is, /tmp/var/ be changed to the owner deployer recursively so that following scp can have access permission. I will see if having time to wire up a PR to fix it.
Thanks for help on this. Ben
Eugen Block <eblock@nde.ag> 于2023年5月8日周一 21:01写道：
Hi,
could you provide some more details about your host OS? Which cephadm version is it? I was able to bootstrap a one-node cluster with both 17.2.5 and 17.2.6 with a non-root user with no such error on openSUSE Leap 15.4:
quincy:~ # rpm -qa | grep cephadm cephadm-17.2.6.248+gad656d572cb-lp154.2.1.noarch
deployer@quincy:~> sudo cephadm --image quay.io/ceph/ceph:v17.2.5 bootstrap --mon-ip 172.17.2.3 --skip-monitoring-stack --ssh-user deployer --single-host-defaults Verifying ssh connectivity ... Adding key to deployer@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... podman (/usr/bin/podman) version 4.4.4 is present [...] Ceph version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available [...] Adding key to deployer@localhost authorized_keys... Adding host quincy... Deploying mon service with default placement... Deploying mgr service with default placement... [...] Bootstrap complete.
Zitat von Ben <ruidong.gao@gmail.com>:
Hi,
with following command:
sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below:
debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write
szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e:
scp:
/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new:
Permission denied
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file
await asyncssh.scp(f.name, (conn, tmp_path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
await source.run(srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
await self._send_files(path, b'')
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
await self._send_file(srcpath, dstpath, attrs)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
await self._make_cd_request(b'C', attrs, size, srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
self._fs.basename(path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
raise exc
Any ideas on this?
Thanks, Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Curiously what is the umask and directory permission in your case? add a host for the cluster for further try? Eugen Block <eblock@nde.ag> 于2023年5月9日周二 14:59写道：
Hi,
I just retried without the single-host option and it worked. Also everything under /tmp/var belongs to root in my case. Unfortunately, I can't use the curl-based cephadm but the contents are identical, I compared. Not sure what it could be at the moment.
Zitat von Ben <ruidong.gao@gmail.com>:
Hi, It is uos v20(with kernel 4.19), one linux distribution among others. no matter since cephadm deploys things in containers by default. cephadm is pulled by curl from Quincy branch of github.
I think you could see some sort of errors if you remove parameter --single-host-defaults.
More investigation shows it looks like a bug with cephadm. during the deploying procedure
,/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new
is created through sudo ssh session remotely(with owner of root) and /tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/ is changed to owner of ssh user deployer. The correct thing to do instead is, /tmp/var/ be changed to the owner deployer recursively so that following scp can have access permission. I will see if having time to wire up a PR to fix it.
Thanks for help on this. Ben
Eugen Block <eblock@nde.ag> 于2023年5月8日周一 21:01写道：
Hi,
could you provide some more details about your host OS? Which cephadm version is it? I was able to bootstrap a one-node cluster with both 17.2.5 and 17.2.6 with a non-root user with no such error on openSUSE Leap 15.4:
quincy:~ # rpm -qa | grep cephadm cephadm-17.2.6.248+gad656d572cb-lp154.2.1.noarch
deployer@quincy:~> sudo cephadm --image quay.io/ceph/ceph:v17.2.5 bootstrap --mon-ip 172.17.2.3 --skip-monitoring-stack --ssh-user deployer --single-host-defaults Verifying ssh connectivity ... Adding key to deployer@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... podman (/usr/bin/podman) version 4.4.4 is present [...] Ceph version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available [...] Adding key to deployer@localhost authorized_keys... Adding host quincy... Deploying mon service with default placement... Deploying mgr service with default placement... [...] Bootstrap complete.
Zitat von Ben <ruidong.gao@gmail.com>:
Hi,
with following command:
sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below:
debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write
szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e:
scp:
/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new:
Permission denied
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file
await asyncssh.scp(f.name, (conn, tmp_path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
await source.run(srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
await self._send_files(path, b'')
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
await self._send_file(srcpath, dstpath, attrs)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
await self._make_cd_request(b'C', attrs, size, srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
self._fs.basename(path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
raise exc
Any ideas on this?
Thanks, Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I used the default to create a new user, so umask is 022. And the /tmp/var/lib/ceph directory belongs to the root user. I haven't tried to add another host yet, I understood that in your case it already failed during the initial boostrap, but I can try to add one more host. Zitat von Ben <ruidong.gao@gmail.com>:
Curiously what is the umask and directory permission in your case? add a host for the cluster for further try?
Eugen Block <eblock@nde.ag> 于2023年5月9日周二 14:59写道：
Hi,
I just retried without the single-host option and it worked. Also everything under /tmp/var belongs to root in my case. Unfortunately, I can't use the curl-based cephadm but the contents are identical, I compared. Not sure what it could be at the moment.
Zitat von Ben <ruidong.gao@gmail.com>:
Hi, It is uos v20(with kernel 4.19), one linux distribution among others. no matter since cephadm deploys things in containers by default. cephadm is pulled by curl from Quincy branch of github.
I think you could see some sort of errors if you remove parameter --single-host-defaults.
More investigation shows it looks like a bug with cephadm. during the deploying procedure
,/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new
is created through sudo ssh session remotely(with owner of root) and /tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/ is changed to owner of ssh user deployer. The correct thing to do instead is, /tmp/var/ be changed to the owner deployer recursively so that following scp can have access permission. I will see if having time to wire up a PR to fix it.
Thanks for help on this. Ben
Eugen Block <eblock@nde.ag> 于2023年5月8日周一 21:01写道：
Hi,
could you provide some more details about your host OS? Which cephadm version is it? I was able to bootstrap a one-node cluster with both 17.2.5 and 17.2.6 with a non-root user with no such error on openSUSE Leap 15.4:
quincy:~ # rpm -qa | grep cephadm cephadm-17.2.6.248+gad656d572cb-lp154.2.1.noarch
deployer@quincy:~> sudo cephadm --image quay.io/ceph/ceph:v17.2.5 bootstrap --mon-ip 172.17.2.3 --skip-monitoring-stack --ssh-user deployer --single-host-defaults Verifying ssh connectivity ... Adding key to deployer@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... podman (/usr/bin/podman) version 4.4.4 is present [...] Ceph version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available [...] Adding key to deployer@localhost authorized_keys... Adding host quincy... Deploying mon service with default placement... Deploying mgr service with default placement... [...] Bootstrap complete.
Zitat von Ben <ruidong.gao@gmail.com>:
Hi,
with following command:
sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below:
debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write
szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e:
scp:
/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new:
Permission denied
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file
await asyncssh.scp(f.name, (conn, tmp_path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
await source.run(srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
await self._send_files(path, b'')
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
await self._send_file(srcpath, dstpath, attrs)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
await self._make_cd_request(b'C', attrs, size, srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
self._fs.basename(path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
raise exc
Any ideas on this?
Thanks, Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Adding a second host worked as well after adding the ceph.pub key to the authorized_keys of the "deployer" user. Zitat von Eugen Block <eblock@nde.ag>:
I used the default to create a new user, so umask is 022. And the /tmp/var/lib/ceph directory belongs to the root user. I haven't tried to add another host yet, I understood that in your case it already failed during the initial boostrap, but I can try to add one more host.
Zitat von Ben <ruidong.gao@gmail.com>:
Curiously what is the umask and directory permission in your case? add a host for the cluster for further try?
Eugen Block <eblock@nde.ag> 于2023年5月9日周二 14:59写道：
Hi,
I just retried without the single-host option and it worked. Also everything under /tmp/var belongs to root in my case. Unfortunately, I can't use the curl-based cephadm but the contents are identical, I compared. Not sure what it could be at the moment.
Zitat von Ben <ruidong.gao@gmail.com>:
Hi, It is uos v20(with kernel 4.19), one linux distribution among others. no matter since cephadm deploys things in containers by default. cephadm is pulled by curl from Quincy branch of github.
I think you could see some sort of errors if you remove parameter --single-host-defaults.
More investigation shows it looks like a bug with cephadm. during the deploying procedure
,/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new
is created through sudo ssh session remotely(with owner of root) and /tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/ is changed to owner of ssh user deployer. The correct thing to do instead is, /tmp/var/ be changed to the owner deployer recursively so that following scp can have access permission. I will see if having time to wire up a PR to fix it.
Thanks for help on this. Ben
Eugen Block <eblock@nde.ag> 于2023年5月8日周一 21:01写道：
Hi,
could you provide some more details about your host OS? Which cephadm version is it? I was able to bootstrap a one-node cluster with both 17.2.5 and 17.2.6 with a non-root user with no such error on openSUSE Leap 15.4:
quincy:~ # rpm -qa | grep cephadm cephadm-17.2.6.248+gad656d572cb-lp154.2.1.noarch
deployer@quincy:~> sudo cephadm --image quay.io/ceph/ceph:v17.2.5 bootstrap --mon-ip 172.17.2.3 --skip-monitoring-stack --ssh-user deployer --single-host-defaults Verifying ssh connectivity ... Adding key to deployer@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... podman (/usr/bin/podman) version 4.4.4 is present [...] Ceph version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available [...] Adding key to deployer@localhost authorized_keys... Adding host quincy... Deploying mon service with default placement... Deploying mgr service with default placement... [...] Bootstrap complete.
Zitat von Ben <ruidong.gao@gmail.com>:
Hi,
with following command:
sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below:
debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write
szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e:
scp:
/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new:
Permission denied
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file
await asyncssh.scp(f.name, (conn, tmp_path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
await source.run(srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
await self._send_files(path, b'')
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
await self._send_file(srcpath, dstpath, attrs)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
await self._make_cd_request(b'C', attrs, size, srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
self._fs.basename(path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
raise exc
Any ideas on this?
Thanks, Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
What's the umask for the "deployer" user? We saw an instance of someone hitting something like this, but for them it seemed to only happen when they had changed the umask to 027. We had patched in https://github.com/ceph/ceph/pull/50736 to address it, which I don't think was merged too late for the 17.2.6 release. On Mon, May 8, 2023 at 5:24 AM Ben <ruidong.gao@gmail.com> wrote:
Hi,
with following command:
sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below:
debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write
szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e: scp:
/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new: Permission denied
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file
await asyncssh.scp(f.name, (conn, tmp_path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
await source.run(srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
await self._send_files(path, b'')
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
await self._send_file(srcpath, dstpath, attrs)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
await self._make_cd_request(b'C', attrs, size, srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
self._fs.basename(path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
raise exc
Any ideas on this?
Thanks, Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
which I think was merged too late* (as in the patch wouldn't be in 17.2.6) On Tue, May 9, 2023 at 5:52 PM Adam King <adking@redhat.com> wrote:
What's the umask for the "deployer" user? We saw an instance of someone hitting something like this, but for them it seemed to only happen when they had changed the umask to 027. We had patched in https://github.com/ceph/ceph/pull/50736 to address it, which I don't think was merged too late for the 17.2.6 release.
On Mon, May 8, 2023 at 5:24 AM Ben <ruidong.gao@gmail.com> wrote:
Hi,
with following command:
sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below:
debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write
szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e: scp:
/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new: Permission denied
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file
await asyncssh.scp(f.name, (conn, tmp_path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
await source.run(srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
await self._send_files(path, b'')
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
await self._send_file(srcpath, dstpath, attrs)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
await self._make_cd_request(b'C', attrs, size, srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
self._fs.basename(path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
raise exc
Any ideas on this?
Thanks, Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
umask here is 027. The pr should fix the problem above. no more fix needed and wait for point release. Adam King <adking@redhat.com> 于2023年5月10日周三 05:52写道：
What's the umask for the "deployer" user? We saw an instance of someone hitting something like this, but for them it seemed to only happen when they had changed the umask to 027. We had patched in https://github.com/ceph/ceph/pull/50736 to address it, which I don't think was merged too late for the 17.2.6 release.
On Mon, May 8, 2023 at 5:24 AM Ben <ruidong.gao@gmail.com> wrote:
Hi,
with following command:
sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below:
debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write
szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e: scp:
/tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new: Permission denied
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file
await asyncssh.scp(f.name, (conn, tmp_path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp
await source.run(srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run
await self._send_files(path, b'')
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files
self.handle_error(exc)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error
raise exc from None
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files
await self._send_file(srcpath, dstpath, attrs)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file
await self._make_cd_request(b'C', attrs, size, srcpath)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request
self._fs.basename(path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request
raise exc
Any ideas on this?
Thanks, Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Adam King
-
Ben
-
Eugen Block