I'm trying to use Object Gateway of Ceph (reef 18.2.1) in conjunction with Openstack Keystone authentication. I created a Keystone account and EC2 Credentail, checked the Access Key and Secret key, and set up AWS credentials. To use the bucket notification feature, I used the asw sns (Simple notification service) cli command with AWS credentials. When requesting rgw instance as an endpoint, a SignatureDoesNotMatch error occurs during the keystone authentication request process. Do you have any experience with similar cases or advice on what to do about related errors? What's odd is that after requesting aws s3api once for that keystone user, if I request the same aws sns command, I receive a normal response. In this process, we confirmed that we were not requesting keystone authentication, but rather using the local token value saved after the s3api request. Below are the log contents and settings. <Log contents> Version information ## openstack root@jsy-ceph-dev:~# openstack --version openstack 5.8.0 ## keystone root@jsy-ceph-dev:~# dpkg -l | grep keystone ii keystone 2:21.0.1-0ubuntu1 all OpenStack identity service - Daemons ii keystone-common 2:21.0.1-0ubuntu1 all OpenStack identity service - Common files ii python3-keystone 2:21.0.1-0ubuntu1 all OpenStack identity service - Python 3 library ii python3-keystoneauth1 4.4.0-0ubuntu1 all authentication library for OpenStack Identity - Python 3.x ii python3-keystoneclient 1:4.4.0-0ubuntu1 all client library for the OpenStack Keystone API - Python 3.x ii python3-keystonemiddleware 9.4.0-0ubuntu1.1 all Middleware for OpenStack Identity (Keystone) - Python 3.x ## ceph ceph version 18.2.1 ## aws-cli root@jsy-ceph-dev:/home/jsy/workspace/test# aws --version aws-cli/2.22.35 Python/3.12.6 Linux/5.15.0-151-generic exe/x86_64.ubuntu.22 <Keystone Log> 2025-08-25 09:47:13.357 80003 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-d00ca94c-122c-42d6-b1cd-c021566d54e8 d8952e6e7a184df59e0f0fbe5f3970b5 e7e897d5419543bda87434921c53bb5b - default default] Authenticating user token process_request /usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py:406 2025-08-25 09:47:13.358 80003 DEBUG keystone.common.fernet_utils [req-d00ca94c-122c-42d6-b1cd-c021566d54e8 d8952e6e7a184df59e0f0fbe5f3970b5 e7e897d5419543bda87434921c53bb5b - default default] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys /usr/lib/python3/dist-packages/keystone/common/fernet_utils.py:286 2025-08-25 09:47:13.417 80003 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-d00ca94c-122c-42d6-b1cd-c021566d54e8 d8952e6e7a184df59e0f0fbe5f3970b5 e7e897d5419543bda87434921c53bb5b - default default] Validating token access rules against request validate_allowed_request /usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py:544 2025-08-25 09:47:13.460 80003 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-93d75d51-2fa6-4050-af83-77fda8ee1ef6 d8952e6e7a184df59e0f0fbe5f3970b5 e7e897d5419543bda87434921c53bb5b - default default] RBAC: auth_context: {'token': <TokenModel (audit_id=Htb3myppTceBAFmMaqxR8g, audit_chain_id=['Htb3myppTceBAFmMaqxR8g']) at 0x7f96e2a96a70>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8952e6e7a184df59e0f0fbe5f3970b5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e7e897d5419543bda87434921c53bb5b', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /usr/lib/python3/dist-packages/keystone/server/flask/request_processing/middleware/auth_context.py:483 2025-08-25 09:47:13.461 80003 DEBUG keystone.server.flask.request_processing.req_logging [req-93d75d51-2fa6-4050-af83-77fda8ee1ef6 d8952e6e7a184df59e0f0fbe5f3970b5 e7e897d5419543bda87434921c53bb5b - default default] REQUEST_METHOD: `POST` log_request_info /usr/lib/python3/dist-packages/keystone/server/flask/request_processing/req_logging.py:27 2025-08-25 09:47:13.461 80003 DEBUG keystone.server.flask.request_processing.req_logging [req-93d75d51-2fa6-4050-af83-77fda8ee1ef6 d8952e6e7a184df59e0f0fbe5f3970b5 e7e897d5419543bda87434921c53bb5b - default default] SCRIPT_NAME: `` log_request_info /usr/lib/python3/dist-packages/keystone/server/flask/request_processing/req_logging.py:28 2025-08-25 09:47:13.461 80003 DEBUG keystone.server.flask.request_processing.req_logging [req-93d75d51-2fa6-4050-af83-77fda8ee1ef6 d8952e6e7a184df59e0f0fbe5f3970b5 e7e897d5419543bda87434921c53bb5b - default default] PATH_INFO: `/v3/s3tokens` log_request_info /usr/lib/python3/dist-packages/keystone/server/flask/request_processing/req_logging.py:29 2025-08-25 09:47:13.466 80003 WARNING keystone.server.flask.application [req-93d75d51-2fa6-4050-af83-77fda8ee1ef6 d8952e6e7a184df59e0f0fbe5f3970b5 e7e897d5419543bda87434921c53bb5b - default default] Authorization failed. The request you have made requires authentication. from 70.60.31.95: keystone.exception.Unauthorized: The request you have made requires authentication. <aws sns debug log> root@jsy-ceph-dev:/home/jsy/workspace/test# aws sns list-topics --endpoint-url=http://192.168.105.11:8088 --profile user_admin --debug 2025-08-25 09:34:10,990 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.22.35 Python/3.12.6 Linux/5.15.0-151-generic exe/x86_64.ubuntu.22 2025-08-25 09:34:10,990 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['sns', 'list-topics', '--endpoint-url=http://192.168.105.11:8088', '--profile', 'user_admin', '--debug'] 2025-08-25 09:34:11,007 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7faec26151c0> 2025-08-25 09:34:11,007 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7faec2840220> 2025-08-25 09:34:11,007 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>> 2025-08-25 09:34:11,008 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7faec299d6c0> 2025-08-25 09:34:11,008 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7faec299eb60> 2025-08-25 09:34:11,008 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7faec2617a60> 2025-08-25 09:34:11,008 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7faec288ae80> 2025-08-25 09:34:11,008 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>> 2025-08-25 09:34:11,008 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7faec2617920> 2025-08-25 09:34:11,008 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7faec26e0590>> 2025-08-25 09:34:11,008 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.22.35/dist/awscli/data/cli.json 2025-08-25 09:34:11,010 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7faec2744360> 2025-08-25 09:34:11,011 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7faec2744680> 2025-08-25 09:34:11,011 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7faec27445e0> 2025-08-25 09:34:11,011 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7faec27447c0> 2025-08-25 09:34:11,011 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7faec2744720> 2025-08-25 09:34:11,011 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7faec26ed800> 2025-08-25 09:34:11,011 - MainThread - botocore.session - DEBUG - Setting config variable for profile to 'user_admin' 2025-08-25 09:34:11,012 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.22.35 Python/3.12.6 Linux/5.15.0-151-generic exe/x86_64.ubuntu.22 2025-08-25 09:34:11,012 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['sns', 'list-topics', '--endpoint-url=http://192.168.105.11:8088', '--profile', 'user_admin', '--debug'] 2025-08-25 09:34:11,012 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7faec2615a80> 2025-08-25 09:34:11,012 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7faec3c42520> 2025-08-25 09:34:11,012 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7faec26a3ba0> 2025-08-25 09:34:11,012 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7faec37c3b00> 2025-08-25 09:34:11,012 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7faec2c56840> 2025-08-25 09:34:11,014 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/ 2025-08-25 09:34:11,015 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set. 2025-08-25 09:34:11,015 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7faec2877600> 2025-08-25 09:34:11,015 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7faec29e9ee0> 2025-08-25 09:34:11,032 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.22.35/dist/awscli/botocore/data/sns/2010-03-31/service-2.json 2025-08-25 09:34:11,036 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sns: calling handler <function add_waiters at 0x7faec2617920> 2025-08-25 09:34:11,052 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sns: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7faec26e0590>> 2025-08-25 09:34:11,053 - MainThread - awscli.clidriver - DEBUG - OrderedDict({'next-token': <awscli.arguments.CLIArgument object at 0x7faec1ddd490>}) 2025-08-25 09:34:11,053 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sns.list-topics: calling handler <function add_streaming_output_arg at 0x7faec2615ee0> 2025-08-25 09:34:11,053 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sns.list-topics: calling handler <function add_cli_input_json at 0x7faec2c571a0> 2025-08-25 09:34:11,054 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sns.list-topics: calling handler <function add_cli_input_yaml at 0x7faec2c57100> 2025-08-25 09:34:11,054 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sns.list-topics: calling handler <function unify_paging_params at 0x7faec2840860> 2025-08-25 09:34:11,069 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.22.35/dist/awscli/botocore/data/sns/2010-03-31/paginators-1.json 2025-08-25 09:34:11,069 - MainThread - awscli.customizations.paginate - DEBUG - Modifying paging parameters for operation: ListTopics 2025-08-25 09:34:11,070 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.sns.list-topics: calling handler <function add_generate_skeleton at 0x7faec272ab60> 2025-08-25 09:34:11,070 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sns.list-topics: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7faec1ddd5e0>> 2025-08-25 09:34:11,070 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sns.list-topics: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7faec1ddd5b0>> 2025-08-25 09:34:11,070 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.sns.list-topics: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7faec1ddf080>> 2025-08-25 09:34:11,070 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sns_list-topics: calling handler <function add_waiters at 0x7faec2617920> 2025-08-25 09:34:11,070 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sns_list-topics: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7faec26e0590>> 2025-08-25 09:34:11,071 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.sns.list-topics: calling handler functools.partial(<function check_should_enable_pagination at 0x7faec28409a0>, ['next-token'], {}, OrderedDict({'next-token': <awscli.arguments.CLIArgument object at 0x7faec1ddd490>, 'cli-input-json': <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7faec1ddd5e0>, 'cli-input-yaml': <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7faec1ddd5b0>, 'starting-token': <awscli.customizations.paginate.PageArgument object at 0x7faec1ddef90>, 'max-items': <awscli.customizations.paginate.PageArgument object at 0x7faec1ddef60>, 'generate-cli-skeleton': <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7faec1ddf080>})) 2025-08-25 09:34:11,071 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sns.list-topics.next-token: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7faec1d252b0> 2025-08-25 09:34:11,071 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sns.list-topics.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7faec1d252b0> 2025-08-25 09:34:11,071 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sns.list-topics.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7faec1d252b0> 2025-08-25 09:34:11,071 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sns.list-topics.starting-token: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7faec1d252b0> 2025-08-25 09:34:11,072 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sns.list-topics.max-items: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7faec1d252b0> 2025-08-25 09:34:11,072 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.sns.list-topics.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7faec1d252b0> 2025-08-25 09:34:11,072 - MainThread - botocore.hooks - DEBUG - Event calling-command.sns.list-topics: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7faec1ddd5e0>> 2025-08-25 09:34:11,072 - MainThread - botocore.hooks - DEBUG - Event calling-command.sns.list-topics: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7faec1ddd5b0>> 2025-08-25 09:34:11,072 - MainThread - botocore.hooks - DEBUG - Event calling-command.sns.list-topics: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7faec1ddf080>> 2025-08-25 09:34:11,072 - MainThread - botocore.hooks - DEBUG - Event calling-command.sns.list-topics: calling handler functools.partial(<function check_should_enable_pagination_call_parameters at 0x7faec2840e00>, ['NextToken']) 2025-08-25 09:34:11,072 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role 2025-08-25 09:34:11,073 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity 2025-08-25 09:34:11,073 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso 2025-08-25 09:34:11,073 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file 2025-08-25 09:34:11,073 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials 2025-08-25 09:34:11,076 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.22.35/dist/awscli/botocore/data/endpoints.json 2025-08-25 09:34:11,102 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7faec4325f80> 2025-08-25 09:34:11,118 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.22.35/dist/awscli/botocore/data/sns/2010-03-31/endpoint-rule-set-1.json 2025-08-25 09:34:11,119 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.22.35/dist/awscli/botocore/data/partitions.json 2025-08-25 09:34:11,120 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.sns: calling handler <function add_generate_presigned_url at 0x7faec499f4c0> 2025-08-25 09:34:11,121 - MainThread - botocore.regions - DEBUG - Creating a regex based endpoint for sns, RegionOne 2025-08-25 09:34:11,123 - MainThread - botocore.endpoint - DEBUG - Setting sns timeout as (60, 60) 2025-08-25 09:34:11,125 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.sns.ListTopics: calling handler <function base64_decode_input_blobs at 0x7faec26b0540> 2025-08-25 09:34:11,125 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.sns.ListTopics: calling handler <function generate_idempotent_uuid at 0x7faec4326480> 2025-08-25 09:34:11,125 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Region': 'RegionOne', 'UseDualStack': False, 'UseFIPS': False, 'Endpoint': 'http://192.168.105.11:8088'} 2025-08-25 09:34:11,125 - MainThread - botocore.regions - DEBUG - Endpoint provider result: http://192.168.105.11:8088 2025-08-25 09:34:11,126 - MainThread - botocore.hooks - DEBUG - Event before-call.sns.ListTopics: calling handler <function add_query_compatibility_header at 0x7faec4340e00> 2025-08-25 09:34:11,126 - MainThread - botocore.hooks - DEBUG - Event before-call.sns.ListTopics: calling handler <function inject_api_version_header_if_needed at 0x7faec4327f60> 2025-08-25 09:34:11,126 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListTopics) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/2.22.35 md/awscrt#0.23.4 ua/2.0 os/linux#5.15.0-151-generic md/arch#x86_64 lang/python#3.12.6 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/distrib#ubuntu.22 md/prompt#off md/command#sns.list-topics'}, 'body': {'Action': 'ListTopics', 'Version': '2010-03-31'}, 'url': 'http://192.168.105.11:8088/', 'context': {'client_region': 'RegionOne', 'client_config': <botocore.config.Config object at 0x7faec16a2f60>, 'has_streaming_input': False, 'auth_type': None, 'unsigned_payload': None}} 2025-08-25 09:34:11,126 - MainThread - botocore.hooks - DEBUG - Event request-created.sns.ListTopics: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7faec41ab620>> 2025-08-25 09:34:11,126 - MainThread - botocore.hooks - DEBUG - Event choose-signer.sns.ListTopics: calling handler <function set_operation_specific_signer at 0x7faec43262a0> 2025-08-25 09:34:11,127 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth. 2025-08-25 09:34:11,127 - MainThread - botocore.auth - DEBUG - CanonicalRequest: POST / content-type:application/x-www-form-urlencoded; charset=utf-8 host:192.168.105.11:8088 x-amz-date:20250825T093411Z content-type;host;x-amz-date 55e772bd3d832d8507965af84ab07fd17d9be3cbb788151ceb0d7fc2be442a54 2025-08-25 09:34:11,127 - MainThread - botocore.auth - DEBUG - StringToSign: AWS4-HMAC-SHA256 20250825T093411Z 20250825/RegionOne/sns/aws4_request 35414dae10a9559317fad3d0adec6b06d75826d8df68caf7b05a97fb3f0ffd41 2025-08-25 09:34:11,127 - MainThread - botocore.auth - DEBUG - Signature: 088516f1cc6e492dc08ff47da44119590ba7977b1269f35f5bd563bceec53674 2025-08-25 09:34:11,127 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=http://192.168.105.11:8088/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.22.35 md/awscrt#0.23.4 ua/2.0 os/linux#5.15.0-151-generic md/arch#x86_64 lang/python#3.12.6 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/distrib#ubuntu.22 md/prompt#off md/command#sns.list-topics', 'X-Amz-Date': b'20250825T093411Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=246edcb3605a4b13946ca04d110a3caa/20250825/RegionOne/sns/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=088516f1cc6e492dc08ff47da44119590ba7977b1269f35f5bd563bceec53674', 'Content-Length': '36'}> 2025-08-25 09:34:11,128 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 192.168.105.11:8088 2025-08-25 09:34:11,583 - MainThread - urllib3.connectionpool - DEBUG - http://192.168.105.11:8088 "POST / HTTP/1.1" 403 212 2025-08-25 09:34:11,584 - MainThread - botocore.parsers - DEBUG - Response headers: {'Content-Length': '212', 'x-amz-request-id': 'tx000005a31b9888451de11-0068ac2e14-393893-suwon-1', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/xml', 'Date': 'Mon, 25 Aug 2025 09:34:12 GMT', 'Connection': 'Keep-Alive'} 2025-08-25 09:34:11,584 - MainThread - botocore.parsers - DEBUG - Response body: b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message></Message><RequestId>tx000005a31b9888451de11-0068ac2e14-393893-suwon-1</RequestId><HostId>393893-suwon-1-kr</HostId></Error>' 2025-08-25 09:34:11,584 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sns.ListTopics: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7faec16a3c20>> 2025-08-25 09:34:11,584 - MainThread - botocore.retries.standard - DEBUG - Not retrying request. 2025-08-25 09:34:11,584 - MainThread - botocore.hooks - DEBUG - Event after-call.sns.ListTopics: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7faec16a3950>> 2025-08-25 09:34:11,586 - MainThread - awscli.clidriver - DEBUG - Exception caught in main() Traceback (most recent call last): File "awscli/clidriver.py", line 499, in main File "awscli/clidriver.py", line 634, in __call__ File "awscli/clidriver.py", line 837, in __call__ File "awscli/clidriver.py", line 965, in invoke File "awscli/clidriver.py", line 987, in _display_response File "awscli/formatter.py", line 77, in __call__ File "awscli/botocore/paginate.py", line 446, in build_full_result File "awscli/botocore/paginate.py", line 252, in __iter__ File "awscli/botocore/paginate.py", line 329, in _make_request File "awscli/botocore/client.py", line 364, in _api_call File "awscli/botocore/client.py", line 744, in _make_api_call botocore.exceptions.ClientError: An error occurred (Unknown) when calling the ListTopics operation: Unknown An error occurred (Unknown) when calling the ListTopics operation: Unknown <rgw instance log> Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: CONTENT_LENGTH=36 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: CONTENT_TYPE=application/x-www-form-urlencoded; charset=utf-8 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: HTTP_ACCEPT_ENCODING=identity Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: HTTP_AUTHORIZATION=AWS4-HMAC-SHA256 Credential=246edcb3605a4b13946ca04d110a3caa/20250825/RegionOne/sns/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=0d60e9a8709998ed8a8d70fc1cb0ad68efeba00e93aaa1fb0617738a5d404bda Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: HTTP_HOST=192.168.105.11:8088 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: HTTP_USER_AGENT=aws-cli/2.22.35 md/awscrt#0.23.4 ua/2.0 os/linux#5.15.0-151-generic md/arch#x86_64 lang/python#3.12.6 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/distrib#ubuntu.22 md/prompt#off md/command#sns.list-topics Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: HTTP_VERSION=1.1 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: HTTP_X_AMZ_DATE=20250825T094713Z Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: REMOTE_ADDR=192.168.105.10 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: REQUEST_METHOD=POST Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: REQUEST_URI=/ Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: SCRIPT_URI=/ Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: SERVER_PORT=8088 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: ====== starting new request req=0x7f39b3c7f730 ===== Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s initializing for trans_id = tx000005df59976568766ba-0068ac3122-393893-suwon-1 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s rgw api priority: s3=8 s3website=7 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s host=192.168.105.11 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/ Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s meta>> HTTP_X_AMZ_DATE Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s x>> x-amz-date:20250825T094713Z Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s Content of POST: Action=ListTopics&Version=2010-03-31 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s get_handler handler=27RGWHandler_REST_PSTopic_AWS Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s handler=27RGWHandler_REST_PSTopic_AWS Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s getting op 4 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s cache get: name=suwon-1.rgw.log++script.prerequest. : hit (negative entry) Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s sns:pubsub_topics_list scheduling with throttler client=3 cost=1 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s sns:pubsub_topics_list op=17RGWPSListTopicsOp Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s sns:pubsub_topics_list verifying requester Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s sns:pubsub_topics_list rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s sns:pubsub_topics_list rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s sns:pubsub_topics_list rgw::auth::s3::S3AnonymousEngine denied with reason=-1 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s sns:pubsub_topics_list rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::AWSv2ExternalAuthStrategy Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s sns:pubsub_topics_list rgw::auth::s3::AWSv2ExternalAuthStrategy: trying rgw::auth::keystone::EC2Engine Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s v4 signature format = 0d60e9a8709998ed8a8d70fc1cb0ad68efeba00e93aaa1fb0617738a5d404bda Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s v4 credential format = 246edcb3605a4b13946ca04d110a3caa/20250825/RegionOne/sns/aws4_request Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s access key id = 246edcb3605a4b13946ca04d110a3caa Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s credential scope = 20250825/RegionOne/sns/aws4_request Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s canonical headers format = content-type:application/x-www-form-urlencoded; charset=utf-8 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s payload request hash = 55e772bd3d832d8507965af84ab07fd17d9be3cbb788151ceb0d7fc2be442a54 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s canonical request = POST Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s canonical request hash = 0629ae314d4cac4bc5f4a109393cb24922239da71a3b9ec1fbf32357e663adaf Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s string to sign = AWS4-HMAC-SHA256 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s get_auth_data_v4: UNSIGNED-PAYLOAD or other v4 no-completer case Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s sns:pubsub_topics_list No stored secret string, cache miss Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.000000000s sns:pubsub_topics_list found cached admin token Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: sending request to http://controller:5000/v3/s3tokens Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: register_request mgr=0x55e4eab762c0 req_data->id=2, curl_handle=0x55e4f30090a0 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: WARNING: blocking http request Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: link_request req_data=0x55e4f486ed80 req_data->id=2, curl_handle=0x55e4f30090a0 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.112000003s sns:pubsub_topics_list rgw::auth::keystone::EC2Engine rejected with reason=-2027 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.112000003s sns:pubsub_topics_list rgw::auth::s3::AWSv2ExternalAuthStrategy rejected with reason=-2027 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.112000003s sns:pubsub_topics_list rgw::auth::s3::AWSAuthStrategy rejected with reason=-2027 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.112000003s sns:pubsub_topics_list Failed the auth strategy, reason=-2027 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: failed to authorize request Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.112000003s op->ERRORHANDLER: err_no=-2027 new_err_no=-2027 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.112000003s cache get: name=suwon-1.rgw.log++script.postrequest. : hit (negative entry) Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.112000003s sns:pubsub_topics_list op status=0 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: req 6770486348358313658 0.112000003s sns:pubsub_topics_list http status=403 Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: ====== req done req=0x7f39b3c7f730 op status=0 http_status=403 latency=0.112000003s ====== Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: beast: 0x7f39b3c7f730: 192.168.105.10 - - [25/Aug/2025:09:47:14.237 +0000] "POST / HTTP/1.1" 403 212 - "aws-cli/2.22.35 md/awscrt#0.23.4 ua/2.0 os/linux#5.15.0-151-generic md/arch#x86_64 lang/python#3.12.6 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/distrib#ubuntu.22 md/prompt#off md/command#sns.list-topics" - latency=0.112000003s Aug 25 09:47:14 sds-ceph-01 radosgw[754427]: failed to read header: end of stream <RGW settings information> root@sds-ceph-01:~# ceph config get client.rgw | grep -E 's3|keystone' client.rgw advanced rgw_keystone_accepted_admin_roles admin * client.rgw advanced rgw_keystone_accepted_roles member, admin * client.rgw advanced rgw_keystone_admin_domain default * client.rgw advanced rgw_keystone_admin_password sds1234 * client.rgw advanced rgw_keystone_admin_project admin * client.rgw advanced rgw_keystone_admin_user admin * client.rgw advanced rgw_keystone_api_version 3 client.rgw advanced rgw_keystone_implicit_tenants true * client.rgw advanced rgw_keystone_token_cache_size 10000 client.rgw basic rgw_keystone_url http://controller:5000 * client.rgw advanced rgw_s3_auth_order external, local * client.rgw advanced rgw_s3_auth_use_keystone true