- ceph-users - lists.ceph.io

Re: Nothing provides libthrift-0.14.0.so()(64bit)
by Graham Derryberry 10 Oct '23

10 Oct '23

I have just started adding a ceph client on a rocky 9 system to our ceph cluster (we're on quincy 17.2.6) and just discovered that epel 9 now provides thrift-0.15.0-2.el9 not thrift-0.14.0-7.el9 as of June 21 2023. So the Nothing provides libthrift-0.14.0.so()(64bit) error has returned! Recommendations?

2 1

outdated mds slow requests
by Ben 10 Oct '23

10 Oct '23

Hi, It is running 17.2.5. there are slow requests warnings in cluster log. ceph tell mds.5 dump_ops_in_flight, get the following. These look like outdated and clients were k8s pods. There are warning of the kind in other mds as well. How could they be cleaned from warnings safely? Many thanks. { "ops": [ { "description": "peer_request(mds.3:5311742.0 authpin)", "initiated_at": "2023-09-14T12:25:43.092201+0000", "age": 926013.05098558997, "duration": 926013.051015759, "type_data": { "flag_point": "dispatched", "reqid": "mds.3:5311742", "op_type": "peer_request", "leader_info": { "leader": "3" }, "request_info": { "attempt": 0, "op_type": "authpin", "lock_type": 0, "object_info": "0x60001205d6d.head", "srcdnpath": "", "destdnpath": "", "witnesses": "", "has_inode_export": false, "inode_export_v": 0, "op_stamp": "0.000000" }, "events": [ { "time": "2023-09-14T12:25:43.092201+0000", "event": "initiated" }, { "time": "2023-09-14T12:25:43.092202+0000", "event": "throttled" }, { "time": "2023-09-14T12:25:43.092201+0000", "event": "header_read" }, { "time": "2023-09-14T12:25:43.092207+0000", "event": "all_read" }, { "time": "2023-09-14T12:25:43.092218+0000", "event": "dispatched" } ] } }, { "description": "peer_request(mds.3:5311743.0 authpin)", "initiated_at": "2023-09-14T12:25:43.092371+0000", "age": 926013.05081614305, "duration": 926013.05089185503, "type_data": { "flag_point": "dispatched", "reqid": "mds.3:5311743", "op_type": "peer_request", "leader_info": { "leader": "3" }, "request_info": { "attempt": 0, "op_type": "authpin", "lock_type": 0, "object_info": "0x60001205d6d.head", "srcdnpath": "", "destdnpath": "", "witnesses": "", "has_inode_export": false, "inode_export_v": 0, "op_stamp": "0.000000" }, "events": [ { "time": "2023-09-14T12:25:43.092371+0000", "event": "initiated" }, { "time": "2023-09-14T12:25:43.092371+0000", "event": "throttled" }, { "time": "2023-09-14T12:25:43.092371+0000", "event": "header_read" }, { "time": "2023-09-14T12:25:43.092374+0000", "event": "all_read" }, { "time": "2023-09-14T12:25:43.092381+0000", "event": "dispatched" } ] } }, { "description": "peer_request(mds.4:4503615.0 authpin)", "initiated_at": "2023-09-14T13:40:25.150040+0000", "age": 921530.99314722, "duration": 921530.99326053297, "type_data": { "flag_point": "dispatched", "reqid": "mds.4:4503615", "op_type": "peer_request", "leader_info": { "leader": "4" }, "request_info": { "attempt": 0, "op_type": "authpin", "lock_type": 0, "object_info": "0x60001205c4f.head", "srcdnpath": "", "destdnpath": "", "witnesses": "", "has_inode_export": false, "inode_export_v": 0, "op_stamp": "0.000000" }, "events": [ { "time": "2023-09-14T13:40:25.150040+0000", "event": "initiated" }, { "time": "2023-09-14T13:40:25.150040+0000", "event": "throttled" }, { "time": "2023-09-14T13:40:25.150040+0000", "event": "header_read" }, { "time": "2023-09-14T13:40:25.150045+0000", "event": "all_read" }, { "time": "2023-09-14T13:40:25.150053+0000", "event": "dispatched" } ] } }, { "description": "client_request(client.460983:5731820 getattr pAsLsXsFs #0x60001205c4f 2023-09-14T13:40:25.144336+0000 caller_uid=0, caller_gid=0{})", "initiated_at": "2023-09-14T13:40:25.150176+0000", "age": 921530.99301089498, "duration": 921530.99316312897, "type_data": { "flag_point": "failed to authpin, inode is being exported", "reqid": "client.460983:5731820", "op_type": "client_request", "client_info": { "client": "client.460983", "tid": 5731820 }, "events": [ { "time": "2023-09-14T13:40:25.150176+0000", "event": "initiated" }, { "time": "2023-09-14T13:40:25.150177+0000", "event": "throttled" }, { "time": "2023-09-14T13:40:25.150176+0000", "event": "header_read" }, { "time": "2023-09-14T13:40:25.150180+0000", "event": "all_read" }, { "time": "2023-09-14T13:40:25.150186+0000", "event": "dispatched" }, { "time": "2023-09-14T13:40:25.150195+0000", "event": "failed to authpin, inode is being exported" } ] } } ], "num_ops": 4 }

2 4

cephadm, cannot use ECDSA key with quincy
by Paul JURCO 10 Oct '23

10 Oct '23

Resent due to moderation when using web interface. Hi ceph users, We have a few clusters with quincy 17.2.6 and we are preparing to migrate from ceph-deploy to cephadm for better management. We are using Ubuntu20 with latest updates (latest openssh). While testing the migration to cephadm on a test cluster with octopus (v16 latest) we had no issues replacing ceph generated cert/key with our own CA signed certs (ECDSA). After upgrading to quincy the test cluster and test again the migration we cannot add hosts due to the errors below, ssh access errors specified a while ago in a tracker. We use the following type of certs: Type: ecdsa-sha2-nistp384-cert-v01(a)openssh.com user certificate The certificate works everytime when using ssh client from shell to connect to all hosts in the cluster. We do a ceph mgr fail every time we replace cert/key so they are restarted. ----- cephadm logs from mgr ------ Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: 10.10.12.160, port 51870 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 111, in redirect_log Oct 06 09:23:27 ceph-m2 bash[1363]: yield Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 90, in _remote_connection Oct 06 09:23:27 ceph-m2 bash[1363]: preferred_auth=['publickey'], options=ssh_options) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect Oct 06 09:23:27 ceph-m2 bash[1363]: 'Opening SSH connection to') Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect Oct 06 09:23:27 ceph-m2 bash[1363]: await conn.wait_established() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established Oct 06 09:23:27 ceph-m2 bash[1363]: await self._waiter Oct 06 09:23:27 ceph-m2 bash[1363]: asyncssh.misc.PermissionDenied: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: During handling of the above exception, another exception occurred: Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper Oct 06 09:23:27 ceph-m2 bash[1363]: return OrchResult(f(*args, **kwargs)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2810, in apply Oct 06 09:23:27 ceph-m2 bash[1363]: results.append(self._apply(spec)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2558, in _apply Oct 06 09:23:27 ceph-m2 bash[1363]: return self._add_host(cast(HostSpec, spec)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1434, in _add_host Oct 06 09:23:27 ceph-m2 bash[1363]: ip_addr = self._check_valid_addr(spec.hostname, spec.addr) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1415, in _check_valid_addr Oct 06 09:23:27 ceph-m2 bash[1363]: error_ok=True, no_fsid=True)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async Oct 06 09:23:27 ceph-m2 bash[1363]: return self.event_loop.get_result(coro) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result Oct 06 09:23:27 ceph-m2 bash[1363]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result Oct 06 09:23:27 ceph-m2 bash[1363]: return self.__get_result() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result Oct 06 09:23:27 ceph-m2 bash[1363]: raise self._exception Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1361, in _run_cephadm Oct 06 09:23:27 ceph-m2 bash[1363]: await self.mgr.ssh._remote_connection(host, addr) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in _remote_connection Oct 06 09:23:27 ceph-m2 bash[1363]: raise Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ Oct 06 09:23:27 ceph-m2 bash[1363]: self.gen.throw(type, value, traceback) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 123, in redirect_log Oct 06 09:23:27 ceph-m2 bash[1363]: raise HostConnectionError(msg, host, addr) Oct 06 09:23:27 ceph-m2 bash[1363]: cephadm.ssh.HostConnectionError: Failed to connect to ceph-m1 (10.10.10.232). Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: 10.10.12.160, port 51870 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection Oct 06 09:23:27 ceph-m2 bash[1363]: debug 2023-10-06T09:23:27.081+0000 7f78d86d8700 -1 log_channel(cephadm) log [ERR] : Failed to connect to ceph-m1 (10.10.10.232). Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: 10.10.12.160, port 51870 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 111, in redirect_log Oct 06 09:23:27 ceph-m2 bash[1363]: yield Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 90, in _remote_connection Oct 06 09:23:27 ceph-m2 bash[1363]: preferred_auth=['publickey'], options=ssh_options) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect Oct 06 09:23:27 ceph-m2 bash[1363]: 'Opening SSH connection to') Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect Oct 06 09:23:27 ceph-m2 bash[1363]: await conn.wait_established() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established Oct 06 09:23:27 ceph-m2 bash[1363]: await self._waiter Oct 06 09:23:27 ceph-m2 bash[1363]: asyncssh.misc.PermissionDenied: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: During handling of the above exception, another exception occurred: Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper Oct 06 09:23:27 ceph-m2 bash[1363]: return OrchResult(f(*args, **kwargs)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2810, in apply Oct 06 09:23:27 ceph-m2 bash[1363]: results.append(self._apply(spec)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2558, in _apply Oct 06 09:23:27 ceph-m2 bash[1363]: return self._add_host(cast(HostSpec, spec)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1434, in _add_host Oct 06 09:23:27 ceph-m2 bash[1363]: ip_addr = self._check_valid_addr(spec.hostname, spec.addr) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 1415, in _check_valid_addr Oct 06 09:23:27 ceph-m2 bash[1363]: error_ok=True, no_fsid=True)) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async Oct 06 09:23:27 ceph-m2 bash[1363]: return self.event_loop.get_result(coro) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result Oct 06 09:23:27 ceph-m2 bash[1363]: return asyncio.run_coroutine_threadsafe(coro, self._loop).result() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result Oct 06 09:23:27 ceph-m2 bash[1363]: return self.__get_result() Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result Oct 06 09:23:27 ceph-m2 bash[1363]: raise self._exception Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1361, in _run_cephadm Oct 06 09:23:27 ceph-m2 bash[1363]: await self.mgr.ssh._remote_connection(host, addr) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in _remote_connection Oct 06 09:23:27 ceph-m2 bash[1363]: raise Oct 06 09:23:27 ceph-m2 bash[1363]: File "/lib64/python3.6/contextlib.py", line 99, in __exit__ Oct 06 09:23:27 ceph-m2 bash[1363]: self.gen.throw(type, value, traceback) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 123, in redirect_log Oct 06 09:23:27 ceph-m2 bash[1363]: raise HostConnectionError(msg, host, addr) Oct 06 09:23:27 ceph-m2 bash[1363]: cephadm.ssh.HostConnectionError: Failed to connect to ceph-m1 (10.10.10.232). Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: 10.10.12.160, port 51870 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, port 22 Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: Permission denied Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection Oct 06 09:23:27 ceph-m2 bash[1363]: debug 2023-10-06T09:23:27.081+0000 7f78d86d8700 -1 mgr handle_command module 'orchestrator' command handler threw exception: __init__() missing 2 required positional arguments: > Oct 06 09:23:27 ceph-m2 bash[1363]: debug 2023-10-06T09:23:27.093+0000 7f78d86d8700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last): Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command Oct 06 09:23:27 ceph-m2 bash[1363]: return self.handle_command(inbuf, cmd) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command Oct 06 09:23:27 ceph-m2 bash[1363]: return dispatch[cmd['prefix']].call(self, cmd, inbuf) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call Oct 06 09:23:27 ceph-m2 bash[1363]: return self.func(mgr, **kwargs) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in <lambda> Oct 06 09:23:27 ceph-m2 bash[1363]: wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper Oct 06 09:23:27 ceph-m2 bash[1363]: return func(*args, **kwargs) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/module.py", line 356, in _add_host Oct 06 09:23:27 ceph-m2 bash[1363]: return self._apply_misc([s], False, Format.plain) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/module.py", line 1092, in _apply_misc Oct 06 09:23:27 ceph-m2 bash[1363]: raise_if_exception(completion) Oct 06 09:23:27 ceph-m2 bash[1363]: File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in raise_if_exception Oct 06 09:23:27 ceph-m2 bash[1363]: e = pickle.loads(c.serialized_exception) Oct 06 09:23:27 ceph-m2 bash[1363]: TypeError: __init__() missing 2 required positional arguments: 'hostname' and 'addr' ----- cephadm logs from mgr ------ ----- sshd logs DEBUG3 level ------ Oct 6 09:33:09 ceph-m1 sshd[57168]: debug2: input_userauth_request: try method publickey [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug2: userauth_pubkey: valid user root querying public key ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzO------------ [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: userauth_pubkey: test pkalg ecdsa-sha2-nistp384 pkblob ECDSA SHA256:m6Q0ZQVjjDLWxbmCn0hcGQ2---------- [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_key_allowed entering [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_send entering: type 22 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive_expect entering: type 23 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive entering [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive entering Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: monitor_read: checking request 22 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_answer_keyallowed entering Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_answer_keyallowed: key_from_blob: 0x5568f0aa7880 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: temporarily_use_uid: 0/0 (e=0/0) Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: trying public key file /etc/ssh/fake_authorized_keys Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: fd 5 clearing O_NONBLOCK Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: restore_uid: 0/0 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_answer_keyallowed: publickey authentication test: ECDSA key is not allowed Oct 6 09:33:09 ceph-m1 sshd[57168]: Failed publickey for root from 10.10.12.160 port 40854 ssh2: ECDSA SHA256:m6Q0ZQVjjDLWxbmCn0hcGQ24gbpk------------- Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_send entering: type 23 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug2: userauth_pubkey: authenticated 0 pkalg ecdsa-sha2-nistp384 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: user_specific_delay: user specific delay 0.000ms [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: ensure_minimum_time_since: elapsed 8.263ms, delaying 8.080ms (requested 8.171ms) [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: userauth_finish: failure partial=0 next methods="publickey" [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: send packet: type 51 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: Connection closed by authenticating user root 10.10.12.160 port 40854 [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: do_cleanup [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: PAM: sshpam_thread_cleanup entering [preauth] Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: monitor_read_log: child log fd closed Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive entering Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: do_cleanup Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: PAM: cleanup Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: PAM: sshpam_thread_cleanup entering Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: Killing privsep child 57169 Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: audit_event: unhandled event 12 Oct 6 09:33:09 ceph-m1 sshd[757]: debug1: main_sigchld_handler: Child exited --------------- I get "ECDSA key is not allowed" above. From sshd logs, it looks like the client is not sending what is required or in the expected format. Now, what was changed in quincy/mgr on ssh client? Is anyone else using ECDSA keys and it works with quincy? I could not find in PRs something specific to this that could block the access, but it might be. Any suggestion? Thank you! Paul

2 2

[RGW] Is there a way for a user to change is secret key or create other keys ?
by Gilles Mocellin 09 Oct '23

09 Oct '23

Hello Cephers, I was using Ceph with OpenStack, and users could add, remove credentials with `openstack ec2 credentials` commands. But, we are moving our Object Storage service to a new cluster, and didn't want to tie it with OpenStack. Is there a way to have a bit of self service for Rados Gateway, at leas for creating, deleting, changing S3 keys ? It does not seem to be part of S3 APIs. It's certainly doable with Ceph RGW admin API, but with which tool that a standard user can use ? The Ceph Dashboard does not seem a good idea. Roles are global, nothing that can be scoped to a tenant. Some S3 browsers exist (https://github.com/nimbis/s3commander) but never with some management like changing S3 keys. Certainly because it's not in the "standard" S3 API. Perhaps Ceph can provide a client-side dashboard, which can be exposed externally, aside the actual admin dashboard, which will stay inside ? Regards, -- Gilles

2 2

Next quincy point release 17.2.7
by Yuri Weinstein 09 Oct '23

09 Oct '23

Hello We are getting very close to the next Quincy point release 17.2.7 Here is the list of must-have PRs https://pad.ceph.com/p/quincy_17.2.7_prs We will start the release testing/review/approval process as soon as all PRs from this list are merged. If you see something missing please speak up and the dev leads will make a decision on including it in this release. TIA

5 5

Random issues with Reef
by Martin Conway 09 Oct '23

09 Oct '23

Hi I have been using Ceph for many years now, and recently upgraded to Reef. Seems I made the jump too quickly, as I have been hitting a few issues. I can't find any mention of them in the bug reports. I thought I would share them here in case it is something to do with my setup. On V18.2.0 cephadm version Fails with the following output: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/sbin/cephadm/__main__.py", line 10096, in <module> File "/usr/sbin/cephadm/__main__.py", line 10084, in main File "/usr/sbin/cephadm/__main__.py", line 2240, in _infer_image File "/usr/sbin/cephadm/__main__.py", line 2338, in infer_local_ceph_image File "/usr/sbin/cephadm/__main__.py", line 2301, in get_container_info File "/usr/sbin/cephadm/__main__.py", line 2301, in <listcomp> File "/usr/sbin/cephadm/__main__.py", line 222, in __getattr__ AttributeError: 'CephadmContext' object has no attribute 'fsid' I don't know if it is related, but cephadm adopt --style legacy --name osd.X Tries to use a V15 image which then fails to start after being imported. The OSD in question has an SSD device from block.db if that is relevant. Using the latest head version of cephadm from github let me work around this issue, but the adopted OSDs were running 18.0.0-6603-g6c4ed58a and needed to be upgraded to 18.2.0. Also the command: ceph orch upgrade start -ceph_version v18.2.0 Does not work, it fails to find the right image. From memory I think it tried to pull quay.io/ceph/ceph:v18:v18.2.0 Ceph orch upgrade start quay.io/ceph/ceph:v18.2.0 Does work as expected. Let me know if there is any other information that would be helpful, but I have since worked around these issues and have my ceph back in a happy state. Regards, Martin Conway IT and Digital Media Manager Research School of Physics Australian National University Canberra ACT 2601 +61 2 6125 1599 https://physics.anu.edu.au<https://physics.anu.edu.au/>

2 2

Received signal: Hangup from killall
by Rok Jaklič 09 Oct '23

09 Oct '23

Hi, yesterday we changed RGW from civetweb to beast and at 04:02 RGW stopped working; we had to restart it in the morning. In one rgw log for previous day we can see: 2023-10-06T04:02:01.105+0200 7fb71d45d700 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 3202663) UID: 0 and in the next day log we can see: 2023-10-06T04:02:01.133+0200 7fb71d45d700 -1 received signal: Hangup from (PID: 3202664) UID: 0 and after that no requests came. We had to restart rgw. In ceph.conf we have something like [client.radosgw.ctplmon2] host = ctplmon2 log_file = /var/log/ceph/client.radosgw.ctplmon2.log rgw_dns_name = ctplmon2 rgw_frontends = "beast ssl_endpoint=0.0.0.0:4443 ssl_certificate=..." rgw_max_put_param_size = 15728640 We assume it has something to do with logrotate. /etc/logrotate.d/ceph: /var/log/ceph/*.log { rotate 90 daily compress sharedscripts postrotate killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror || pkill -1 -x "ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw|rbd-mirror|cephfs-mirror" || true endscript missingok notifempty su root ceph } ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) And ideas why this happend? Kind regards, Rok

1 1

Ceph 16.2.x excessive logging, how to reduce?
by Zakhar Kirpichenko 09 Oct '23

09 Oct '23

Hi, Our Ceph 16.2.x cluster managed by cephadm is logging a lot of very detailed messages, Ceph logs alone on hosts with monitors and several OSDs has already eaten through 50% of the endurance of the flash system drives over a couple of years. Cluster logging settings are default, and it seems that all daemons are writing lots and lots of debug information to the logs, such as for example: https://pastebin.com/ebZq8KZk (it's just a snippet, but there's lots and lots of various information). Is there a way to reduce the amount of logging and, for example, limit the logging to warnings or important messages so that it doesn't include every successful authentication attempt, compaction etc, etc, when the cluster is healthy and operating normally? I would very much appreciate your advice on this. Best regards, Zakhar

4 12

Manual resharding with multisite
by Yixin Jin 09 Oct '23

09 Oct '23

Hi folks, I am aware that dynamic resharding isn't supported before Reef with multisite. However, does manual resharding work? It doesn't seem to be so, either. First of all, running "bucket reshard" has to be in the master zone. But if the objects of that bucket isn't in the master zone, resharding in the master zone seems to render those objects inaccessible in the zone that actually has them. So, what is recommended practice of resharding with multiste? No resharding at all? Thanks, Yixin

3 2

If you know your cluster is performing as expected？
by Louis Koo 08 Oct '23

08 Oct '23

My ceph cluster was made of all-flash nvme, like this: 1. 10*nodes , every node has 21 nvme device was used to rgw data pool, and other 1 nvme was used to index pool ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -26 1467.11609 root bd-ssd03-data -43 146.71161 host bd-ssd03-node01-f6fe3bdc-a1b6-496f-8fff-adb240afaedf 0 nvme 6.98627 osd.0 up 1.00000 1.00000 4 nvme 6.98627 osd.4 up 1.00000 1.00000 11 nvme 6.98627 osd.11 up 1.00000 1.00000 21 nvme 6.98627 osd.21 up 1.00000 1.00000 31 nvme 6.98627 osd.31 up 1.00000 1.00000 46 nvme 6.98627 osd.46 up 1.00000 1.00000 56 nvme 6.98627 osd.56 up 1.00000 1.00000 67 nvme 6.98627 osd.67 up 1.00000 1.00000 77 nvme 6.98627 osd.77 up 1.00000 1.00000 87 nvme 6.98627 osd.87 up 1.00000 1.00000 97 nvme 6.98627 osd.97 up 1.00000 1.00000 107 nvme 6.98627 osd.107 up 1.00000 1.00000 117 nvme 6.98627 osd.117 up 1.00000 1.00000 127 nvme 6.98627 osd.127 up 1.00000 1.00000 137 nvme 6.98627 osd.137 up 1.00000 1.00000 147 nvme 6.98627 osd.147 up 1.00000 1.00000 157 nvme 6.98627 osd.157 up 1.00000 1.00000 168 nvme 6.98627 osd.168 up 1.00000 1.00000 178 nvme 6.98627 osd.178 up 1.00000 1.00000 188 nvme 6.98627 osd.188 up 1.00000 1.00000 198 nvme 6.98627 osd.198 up 1.00000 1.00000 -31 146.71161 host bd-ssd03-node02-ffedbbd9-3794-4b10-ac1d-bf7595395892 1 nvme 6.98627 osd.1 up 1.00000 1.00000 6 nvme 6.98627 osd.6 up 1.00000 1.00000 14 nvme 6.98627 osd.14 up 1.00000 1.00000 24 nvme 6.98627 osd.24 up 1.00000 1.00000 34 nvme 6.98627 osd.34 up 1.00000 1.00000 45 nvme 6.98627 osd.45 up 1.00000 1.00000 55 nvme 6.98627 osd.55 up 1.00000 1.00000 66 nvme 6.98627 osd.66 up 1.00000 1.00000 76 nvme 6.98627 osd.76 up 1.00000 1.00000 86 nvme 6.98627 osd.86 up 1.00000 1.00000 96 nvme 6.98627 osd.96 up 1.00000 1.00000 106 nvme 6.98627 osd.106 up 1.00000 1.00000 116 nvme 6.98627 osd.116 up 1.00000 1.00000 130 nvme 6.98627 osd.130 up 1.00000 1.00000 141 nvme 6.98627 osd.141 up 1.00000 1.00000 151 nvme 6.98627 osd.151 up 1.00000 1.00000 161 nvme 6.98627 osd.161 up 1.00000 1.00000 171 nvme 6.98627 osd.171 up 1.00000 1.00000 181 nvme 6.98627 osd.181 up 1.00000 1.00000 191 nvme 6.98627 osd.191 up 1.00000 1.00000 201 nvme 6.98627 osd.201 up 1.00000 1.00000 -29 146.71161 host bd-ssd03-node03-637c96a1-6b93-42cc-9b35-44b729bd7a5c 2 nvme 6.98627 osd.2 up 1.00000 1.00000 8 nvme 6.98627 osd.8 up 1.00000 1.00000 17 nvme 6.98627 osd.17 up 1.00000 1.00000 27 nvme 6.98627 osd.27 up 1.00000 1.00000 37 nvme 6.98627 osd.37 up 1.00000 1.00000 48 nvme 6.98627 osd.48 up 1.00000 1.00000 58 nvme 6.98627 osd.58 up 1.00000 1.00000 68 nvme 6.98627 osd.68 up 1.00000 1.00000 79 nvme 6.98627 osd.79 up 1.00000 1.00000 89 nvme 6.98627 osd.89 up 1.00000 1.00000 99 nvme 6.98627 osd.99 up 1.00000 1.00000 109 nvme 6.98627 osd.109 up 1.00000 1.00000 119 nvme 6.98627 osd.119 up 1.00000 1.00000 129 nvme 6.98627 osd.129 up 1.00000 1.00000 140 nvme 6.98627 osd.140 up 1.00000 1.00000 150 nvme 6.98627 osd.150 up 1.00000 1.00000 166 nvme 6.98627 osd.166 up 1.00000 1.00000 176 nvme 6.98627 osd.176 up 1.00000 1.00000 186 nvme 6.98627 osd.186 up 1.00000 1.00000 196 nvme 6.98627 osd.196 up 1.00000 1.00000 206 nvme 6.98627 osd.206 up 1.00000 1.00000 -41 146.71161 host bd-ssd03-node04-87186f77-3df0-4b71-8103-fdc2a104eb17 3 nvme 6.98627 osd.3 up 1.00000 1.00000 10 nvme 6.98627 osd.10 up 1.00000 1.00000 20 nvme 6.98627 osd.20 up 1.00000 1.00000 30 nvme 6.98627 osd.30 up 1.00000 1.00000 40 nvme 6.98627 osd.40 up 1.00000 1.00000 50 nvme 6.98627 osd.50 up 1.00000 1.00000 60 nvme 6.98627 osd.60 up 1.00000 1.00000 70 nvme 6.98627 osd.70 up 1.00000 1.00000 81 nvme 6.98627 osd.81 up 1.00000 1.00000 90 nvme 6.98627 osd.90 up 1.00000 1.00000 100 nvme 6.98627 osd.100 up 1.00000 1.00000 110 nvme 6.98627 osd.110 up 1.00000 1.00000 120 nvme 6.98627 osd.120 up 1.00000 1.00000 132 nvme 6.98627 osd.132 up 1.00000 1.00000 143 nvme 6.98627 osd.143 up 1.00000 1.00000 153 nvme 6.98627 osd.153 up 1.00000 1.00000 162 nvme 6.98627 osd.162 up 1.00000 1.00000 172 nvme 6.98627 osd.172 up 1.00000 1.00000 182 nvme 6.98627 osd.182 up 1.00000 1.00000 192 nvme 6.98627 osd.192 up 1.00000 1.00000 202 nvme 6.98627 osd.202 up 1.00000 1.00000 -25 146.71161 host bd-ssd03-node05-a6f31379-cb27-4bc3-bbda-726023d9ff96 5 nvme 6.98627 osd.5 up 1.00000 1.00000 13 nvme 6.98627 osd.13 up 1.00000 1.00000 23 nvme 6.98627 osd.23 up 1.00000 1.00000 33 nvme 6.98627 osd.33 up 1.00000 1.00000 43 nvme 6.98627 osd.43 up 1.00000 1.00000 53 nvme 6.98627 osd.53 up 1.00000 1.00000 64 nvme 6.98627 osd.64 up 1.00000 1.00000 78 nvme 6.98627 osd.78 up 1.00000 1.00000 88 nvme 6.98627 osd.88 up 1.00000 1.00000 98 nvme 6.98627 osd.98 up 1.00000 1.00000 108 nvme 6.98627 osd.108 up 1.00000 1.00000 118 nvme 6.98627 osd.118 up 1.00000 1.00000 128 nvme 6.98627 osd.128 up 1.00000 1.00000 139 nvme 6.98627 osd.139 up 1.00000 1.00000 149 nvme 6.98627 osd.149 up 1.00000 1.00000 159 nvme 6.98627 osd.159 up 1.00000 1.00000 169 nvme 6.98627 osd.169 up 1.00000 1.00000 179 nvme 6.98627 osd.179 up 1.00000 1.00000 189 nvme 6.98627 osd.189 up 1.00000 1.00000 199 nvme 6.98627 osd.199 up 1.00000 1.00000 209 nvme 6.98627 osd.209 up 1.00000 1.00000 -33 146.71161 host bd-ssd03-node06-e4288b8f-7d64-40c1-a3f1-c60757e1fbb1 7 nvme 6.98627 osd.7 up 1.00000 1.00000 16 nvme 6.98627 osd.16 up 1.00000 1.00000 26 nvme 6.98627 osd.26 up 1.00000 1.00000 36 nvme 6.98627 osd.36 up 1.00000 1.00000 51 nvme 6.98627 osd.51 up 1.00000 1.00000 62 nvme 6.98627 osd.62 up 1.00000 1.00000 71 nvme 6.98627 osd.71 up 1.00000 1.00000 82 nvme 6.98627 osd.82 up 1.00000 1.00000 93 nvme 6.98627 osd.93 up 1.00000 1.00000 103 nvme 6.98627 osd.103 up 1.00000 1.00000 113 nvme 6.98627 osd.113 up 1.00000 1.00000 123 nvme 6.98627 osd.123 up 1.00000 1.00000 134 nvme 6.98627 osd.134 up 1.00000 1.00000 145 nvme 6.98627 osd.145 up 1.00000 1.00000 155 nvme 6.98627 osd.155 up 1.00000 1.00000 165 nvme 6.98627 osd.165 up 1.00000 1.00000 175 nvme 6.98627 osd.175 up 1.00000 1.00000 185 nvme 6.98627 osd.185 up 1.00000 1.00000 195 nvme 6.98627 osd.195 up 1.00000 1.00000 208 nvme 6.98627 osd.208 up 1.00000 1.00000 215 nvme 6.98627 osd.215 up 1.00000 1.00000 -37 146.71161 host bd-ssd03-node07-fabbd3f5-1466-493b-85f4-dac9254720c4 9 nvme 6.98627 osd.9 up 1.00000 1.00000 18 nvme 6.98627 osd.18 up 1.00000 1.00000 28 nvme 6.98627 osd.28 up 1.00000 1.00000 44 nvme 6.98627 osd.44 up 1.00000 1.00000 54 nvme 6.98627 osd.54 up 1.00000 1.00000 65 nvme 6.98627 osd.65 up 1.00000 1.00000 75 nvme 6.98627 osd.75 up 1.00000 1.00000 85 nvme 6.98627 osd.85 up 1.00000 1.00000 95 nvme 6.98627 osd.95 up 1.00000 1.00000 105 nvme 6.98627 osd.105 up 1.00000 1.00000 115 nvme 6.98627 osd.115 up 1.00000 1.00000 125 nvme 6.98627 osd.125 up 1.00000 1.00000 136 nvme 6.98627 osd.136 up 1.00000 1.00000 146 nvme 6.98627 osd.146 up 1.00000 1.00000 156 nvme 6.98627 osd.156 up 1.00000 1.00000 167 nvme 6.98627 osd.167 up 1.00000 1.00000 177 nvme 6.98627 osd.177 up 1.00000 1.00000 187 nvme 6.98627 osd.187 up 1.00000 1.00000 197 nvme 6.98627 osd.197 up 1.00000 1.00000 207 nvme 6.98627 osd.207 up 1.00000 1.00000 214 nvme 6.98627 osd.214 up 1.00000 1.00000 -35 146.71161 host bd-ssd03-node08-65ab0995-22d0-4b4c-b661-4cc53ceb5b61 12 nvme 6.98627 osd.12 up 1.00000 1.00000 22 nvme 6.98627 osd.22 up 1.00000 1.00000 32 nvme 6.98627 osd.32 up 1.00000 1.00000 42 nvme 6.98627 osd.42 up 1.00000 1.00000 52 nvme 6.98627 osd.52 up 1.00000 1.00000 63 nvme 6.98627 osd.63 up 1.00000 1.00000 73 nvme 6.98627 osd.73 up 1.00000 1.00000 84 nvme 6.98627 osd.84 up 1.00000 1.00000 94 nvme 6.98627 osd.94 up 1.00000 1.00000 104 nvme 6.98627 osd.104 up 1.00000 1.00000 114 nvme 6.98627 osd.114 up 1.00000 1.00000 124 nvme 6.98627 osd.124 up 1.00000 1.00000 138 nvme 6.98627 osd.138 up 1.00000 1.00000 148 nvme 6.98627 osd.148 up 1.00000 1.00000 158 nvme 6.98627 osd.158 up 1.00000 1.00000 170 nvme 6.98627 osd.170 up 1.00000 1.00000 180 nvme 6.98627 osd.180 up 1.00000 1.00000 190 nvme 6.98627 osd.190 up 1.00000 1.00000 200 nvme 6.98627 osd.200 up 1.00000 1.00000 210 nvme 6.98627 osd.210 up 1.00000 1.00000 216 nvme 6.98627 osd.216 up 1.00000 1.00000 -39 146.71161 host bd-ssd03-node09-4833dda0-f95f-4116-97fa-635ee8685ba2 15 nvme 6.98627 osd.15 up 1.00000 1.00000 25 nvme 6.98627 osd.25 up 1.00000 1.00000 35 nvme 6.98627 osd.35 up 1.00000 1.00000 47 nvme 6.98627 osd.47 up 1.00000 1.00000 61 nvme 6.98627 osd.61 up 1.00000 1.00000 72 nvme 6.98627 osd.72 up 1.00000 1.00000 83 nvme 6.98627 osd.83 up 1.00000 1.00000 92 nvme 6.98627 osd.92 up 1.00000 1.00000 102 nvme 6.98627 osd.102 up 1.00000 1.00000 112 nvme 6.98627 osd.112 up 1.00000 1.00000 122 nvme 6.98627 osd.122 up 1.00000 1.00000 133 nvme 6.98627 osd.133 up 1.00000 1.00000 144 nvme 6.98627 osd.144 up 1.00000 1.00000 154 nvme 6.98627 osd.154 up 1.00000 1.00000 164 nvme 6.98627 osd.164 up 1.00000 1.00000 174 nvme 6.98627 osd.174 up 1.00000 1.00000 184 nvme 6.98627 osd.184 up 1.00000 1.00000 194 nvme 6.98627 osd.194 up 1.00000 1.00000 204 nvme 6.98627 osd.204 up 1.00000 1.00000 213 nvme 6.98627 osd.213 up 1.00000 1.00000 218 nvme 6.98627 osd.218 up 1.00000 1.00000 -45 146.71161 host bd-ssd03-node10-9cfe177c-2e0f-43d3-a1a9-6524dac0009a 19 nvme 6.98627 osd.19 up 1.00000 1.00000 29 nvme 6.98627 osd.29 up 1.00000 1.00000 39 nvme 6.98627 osd.39 up 1.00000 1.00000 49 nvme 6.98627 osd.49 up 1.00000 1.00000 59 nvme 6.98627 osd.59 up 1.00000 1.00000 69 nvme 6.98627 osd.69 up 1.00000 1.00000 80 nvme 6.98627 osd.80 up 1.00000 1.00000 91 nvme 6.98627 osd.91 up 1.00000 1.00000 101 nvme 6.98627 osd.101 up 1.00000 1.00000 111 nvme 6.98627 osd.111 up 1.00000 1.00000 121 nvme 6.98627 osd.121 up 1.00000 1.00000 131 nvme 6.98627 osd.131 up 1.00000 1.00000 142 nvme 6.98627 osd.142 up 1.00000 1.00000 152 nvme 6.98627 osd.152 up 1.00000 1.00000 163 nvme 6.98627 osd.163 up 1.00000 1.00000 173 nvme 6.98627 osd.173 up 1.00000 1.00000 183 nvme 6.98627 osd.183 up 1.00000 1.00000 193 nvme 6.98627 osd.193 up 1.00000 1.00000 203 nvme 6.98627 osd.203 up 1.00000 1.00000 212 nvme 6.98627 osd.212 up 1.00000 1.00000 217 nvme 6.98627 osd.217 up 1.00000 1.00000 -4 14.55475 root bd-ssd03-index -11 1.45547 host bd-ssd03-node01-0581ef65-8df6-4cfc-8992-08ffb7959914 41 nvme 1.45547 osd.41 up 1.00000 1.00000 -13 1.45547 host bd-ssd03-node02-0025ee25-77d9-4731-bc94-1c541e1fa2e7 126 nvme 1.45547 osd.126 up 1.00000 1.00000 -3 1.45547 host bd-ssd03-node03-a2629c55-9ddc-4c16-ba85-187af70eef23 160 nvme 1.45547 osd.160 up 1.00000 1.00000 -9 1.45547 host bd-ssd03-node04-9e02d968-28b5-43a8-b93a-418cb61af7ee 211 nvme 1.45547 osd.211 up 1.00000 1.00000 -15 1.45547 host bd-ssd03-node05-536f9f5e-cf15-4bb5-abaa-afd313473ab1 74 nvme 1.45547 osd.74 up 1.00000 1.00000 -23 1.45547 host bd-ssd03-node06-43949775-80b1-4297-b23f-d36ffb784d76 205 nvme 1.45547 osd.205 up 1.00000 1.00000 -21 1.45547 host bd-ssd03-node07-11ca9165-47b2-4b1d-84d7-c6b4f62f51eb 38 nvme 1.45547 osd.38 up 1.00000 1.00000 -7 1.45547 host bd-ssd03-node08-d6739f0c-a356-415c-ae4d-323cb4197efe 135 nvme 1.45547 osd.135 up 1.00000 1.00000 -17 1.45547 host bd-ssd03-node09-ece27fca-7711-45f1-b899-6d6f61aad2cf 57 nvme 1.45547 osd.57 up 1.00000 1.00000 -19 1.45547 host bd-ssd03-node10-c53d891c-b393-4663-88bf-f2ec4e9e0a9a 219 nvme 1.45547 osd.219 up 1.00000 1.00000 2. memory 512G [root@bd-ssd03-node01 ~]# free -g total used free shared buff/cache available Mem: 503 105 134 0 262 393 Swap: 0 0 0 3. cpu 128 Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 256 On-line CPU(s) list: 0-255 Thread(s) per core: 2 Core(s) per socket: 64 Socket(s): 2 NUMA node(s): 2 Vendor ID: AuthenticAMD BIOS Vendor ID: Advanced Micro Devices, Inc. CPU family: 25 Model: 1 Model name: AMD EPYC 7763 64-Core Processor BIOS Model name: AMD EPYC 7763 64-Core Processor Stepping: 1 CPU MHz: 3243.566 CPU max MHz: 3529.0520 CPU min MHz: 1500.0000 BogoMIPS: 4890.70 Virtualization: AMD-V L1d cache: 32K L1i cache: 32K L2 cache: 512K L3 cache: 32768K 4. Network 50Gb/s Settings for bond4: Supported ports: [ ] Supported link modes: Not reported Supported pause frame use: No Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: 50000Mb/s Duplex: Full Auto-negotiation: off Port: Other PHYAD: 0 Transceiver: internal Link detected: yes I'm using cosbench to test the rgw, the xml config of the cosbench is 10 driver to 10 rgw and every driver with 16 works: <?xml version="1.0" encoding="UTF-8" ?> <workload name="test" description="sample benchmark for s3"> <workflow> <workstage name="create bucket" closuredelay="0" config=""> <auth type="none" config=""/> <work name="rgw1-create" type="init" workers="1" interval="5" division="container" totalOps="2560000" rampup="0" rampdown="0" afr="0" totalBytes="0" config="cprefix=bdssd03-bucket;containers=r(1,10)"> <auth type="none" config=""/> <storage type="s3" config="accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.143:80;path_style_access=true"/> <operation type="init" ratio="100" division="container" config="containers=r(1,1);objects=r(0,0);sizes=c(0)B" id="none"/> </work> </workstage> <workstage name="write"> <work name="w1" workers="16" totalOps="2560000" driver="driver1"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(1);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.144:80;server_side_encryption=false; "/> </work> <work name="w2" workers="16" totalOps="2560000" driver="driver2"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(2);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.145:80;server_side_encryption=false;" /> </work> <work name="w3" workers="16" totalOps="2560000" driver="driver3"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(3);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.146:80;server_side_encryption=false;" /> </work> <work name="w4" workers="16" totalOps="2560000" driver="driver4"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(4);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.147:80;server_side_encryption=false; "/> </work> <work name="w5" workers="16" totalOps="2560000" driver="driver5"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(5);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.148:80;server_side_encryption=false;" /> </work> <work name="w6" workers="16" totalOps="2560000" driver="driver6"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(6);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.149:80;server_side_encryption=false;" /> </work> <work name="w7" workers="16" totalOps="2560000" driver="driver7"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(7);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.150:80;server_side_encryption=false; "/> </work> <work name="w8" workers="16" totalOps="2560000" driver="driver8"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(8);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.151:80;server_side_encryption=false;" /> </work> <work name="w9" workers="16" totalOps="2560000" driver="driver9"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(9);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.152:80;server_side_encryption=false;" /> </work> <work name="w10" workers="16" totalOps="2560000" driver="driver10"> <operation type="write" ratio="100" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=c(10);objects=r(1,2000000);sizes=c(4)MB;" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.143:80;server_side_encryption=false; "/> </work> </workstage> <workstage name="cleanup"> <work type="cleanup" workers="16" config="cprefix=bdssd03-bucket;oprefix=test4m-;containers=r(1,10);objects=r(1,2000000)" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.143:80;server_side_encryption=false;" /> </workstage> <workstage name="dispose"> <work type="dispose" workers="10" config="cprefix=bdssd03-bucket;containers=r(1,10)" /> <storage type="s3" config="path_style_access=true;accesskey=GHWT013TX7ISVKR430Y0;secretkey=u7aCkNfznEztWpVJcARHceVC0ZPjR8XiVQDPmzAR;endpoint=http://10.x.x.143:80;server_side_encryption=false;" /> </workstage> </workflow> </workload> the performance result is: cluster: id: 075b70fa-6a72-4ce2-8956-d249c7cf10ce health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 17h) mgr: a(active, since 17h), standbys: b osd: 220 osds: 220 up (since 17h), 220 in (since 17h) rgw: 10 daemons active (10 hosts, 1 zones) data: pools: 9 pools, 4273 pgs objects: 17.76M objects, 65 TiB usage: 82 TiB used, 1.4 PiB / 1.4 PiB avail pgs: 4273 active+clean io: client: 11 MiB/s rd, 9.2 GiB/s wr, 15.44k op/s rd, 28.70k op/s wr only can reach to 9.2 GiB with 4MB write, I don't know that whether the performance meet expectations?

1 0