Re: [RGW] "Failed to reserve notification on queue: error: -22" after upgrading Reef to Tentacle(20.2.0)
Hi Steve, There were several internal format changes between reef and squid. There is a migration process that should kick in once you enable the notification_v2 zone feature. However, everything should still work even if that feature is not enabled. Could you please share a snippet of the rgw debug logs showing more detailed error information? As a last resort, you can use the new "topic dump" command to get all of the notifications in json format (so the information is not lost), and then delete and recreate the topics. Yuval On Tue, Apr 14, 2026 at 8:42 PM steve jung via dev <dev@ceph.io> wrote:
Hello Ceph community,
I am seeking advice regarding an RGW notification issue encountered after upgrading our cluster from Reef to Tentacle.
We heavily use the RGW PubSub (topic/notification) feature. We recently upgraded to Tentacle specifically to leverage new reliability options such as time_to_live, max_retries, and retry_sleep_duration to prevent data loss during transmission.
The Problem:
After the upgrade, when using existing topics created in the Reef version, we started seeing the following error: "Failed to reserve notification on queue: error: -22" Consequently, all PUT requests to the affected buckets are failing with an error. (s3 response include error message "<Code>InvalidArgument</Code> Steps Taken (Troubleshooting):
To resolve this without deleting our existing topics/notifications, we have attempted the following: - Restarted all RGW instances. - Redeployed the RGW gateways. - Performed rados clearomap on the pubsub..bucket.xxxx objects found in the logs. Despite these efforts, the PUT operations continue to fail with the same error. We are trying to find a fix that doesn't involve recreating the topics and notifications, as we are still in the process of debugging the root cause.
Questions:
- Has anyone encountered this -22 (EINVAL) error specifically during a Reef-to-Tentacle transition involving PubSub? - Are there any internal OMAP structure changes in Tentacle that might cause incompatibility with legacy Reef topics? - Any suggestions or ideas on how to clear this "queue reservation" error without losing the current notification configuration? Any insights or guidance would be greatly appreciated.
Best regards,
Hello, Yuval Lifshitz Thank you for your reply. Current status As you suggested, we have evaluated a backup‑then‑delete‑and‑re‑create approach for the topic settings. However, our service relies on automatic creation and management of topics and notifications, making manual deletion/re‑creation burdensome. Consequently, we are keeping the existing topics and notifications while exploring a solution that does not require their removal, and the issue remains open in its current state. Additional information shared 1) s3 put error log --------------------------------------------- aws s3 cp hello.txt s3://test-migration-src-103/hello.txt --endpoint-url=http://192.168.103.132:8088 --profile src --debug (... skipped ...) 2026-04-17 01:26:34,707 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Calculating signature using v4 auth. 2026-04-17 01:26:34,708 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - CanonicalRequest: PUT /test-migration-src-102/hello.txt content-type:text/plain host:192.168.103.132:8088 x-amz-checksum-crc64nvme:9Df1tNKUSe0= x-amz-content-sha256:e0d47bd9d47768eee6fec055a12136893da72db2cf65a8beb105f043c4db43d3 x-amz-date:20260417T012634Z x-amz-sdk-checksum-algorithm:CRC64NVME content-type;host;x-amz-checksum-crc64nvme;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm e0d47bd9d47768eee6fec055a12136893da72db2cf65a8beb105f043c4db43d3 2026-04-17 01:26:34,708 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - StringToSign: AWS4-HMAC-SHA256 20260417T012634Z 20260417/us-east-1/s3/aws4_request e09005c8ed5dc099076296250c75c8230178f45de6c161a12fd9bdb5a830c687 2026-04-17 01:26:34,708 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Signature: df352be04be65e446bbff340ada8de882696f8f43c50b1877ce771fde20a165c 2026-04-17 01:26:34,708 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.PutObject: calling handler <function signal_transferring at 0x7fdec0cf3a60> 2026-04-17 01:26:34,708 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=http://192.168.103.132:8088/test-migration-src-102/hello.txt, headers={'x-amz-sdk-checksum-algorithm': b'CRC64NVME', 'Content-Type': b'text/plain', 'User-Agent': b'aws-cli/2.23.0 md/awscrt#0.23.4 ua/2.0 os/linux#5.15.0-173-generic md/arch#x86_64 lang/python#3.12.6 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/distrib#ubuntu.22 md/prompt#off md/command#s3.cp', 'Expect': b'100-continue', 'x-amz-checksum-crc64nvme': b'9Df1tNKUSe0=', 'X-Amz-Date': b'20260417T012634Z', 'X-Amz-Content-SHA256': b'e0d47bd9d47768eee6fec055a12136893da72db2cf65a8beb105f043c4db43d3', 'Authorization': b'AWS4-HMAC-SHA256 Credential=mw50/20260417/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-checksum-crc64nvme;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm, Signature=df352be04be65e446bbff340ada8de882696f8f43c50b1877ce771fde20a165c', 'Content-Length': '14'}> 2026-04-17 01:26:34,712 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 192.168.103.132:8088 2026-04-17 01:26:34,713 - ThreadPoolExecutor-0_0 - botocore.awsrequest - DEBUG - Waiting for 100 Continue response. 2026-04-17 01:26:34,716 - ThreadPoolExecutor-0_0 - botocore.awsrequest - DEBUG - 100 Continue response seen, now sending request body. 2026-04-17 01:26:34,845 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - http://192.168.103.132:8088 "PUT /test-migration-src-102/hello.txt HTTP/1.1" 400 253 2026-04-17 01:26:34,881 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parse.s3.PutObject: calling handler <function _handle_200_error at 0x7fdec2349120> 2026-04-17 01:26:34,881 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parse.s3.PutObject: calling handler <function handle_expires_header at 0x7fdec2348f40> 2026-04-17 01:26:34,881 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response headers: {'Content-Length': '253', 'x-amz-request-id': 'tx000008c991be4fd8a9af4-0069e18c4a-301775-suwon-1', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/xml', 'Server': 'Ceph Object Gateway (tentacle)', 'Date': 'Fri, 17 Apr 2026 01:26:34 GMT', 'Connection': 'Keep-Alive'} 2026-04-17 01:26:34,881 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response body: b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><Message></Message><BucketName>test-migration-src-102</BucketName><RequestId>tx000008c991be4fd8a9af4-0069e18c4a-301775-suwon-1</RequestId><HostId>301775-suwon-1-kr</HostId></Error>' 2026-04-17 01:26:34,884 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <function _update_status_code at 0x7fdec2349260> 2026-04-17 01:26:34,884 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fdebf340a70>> 2026-04-17 01:26:34,884 - ThreadPoolExecutor-0_0 - botocore.retries.standard - DEBUG - Not retrying request. 2026-04-17 01:26:34,884 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fdebf340800>> 2026-04-17 01:26:34,884 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event after-call.s3.PutObject: calling handler <function enhance_error_msg at 0x7fdec0615ee0> 2026-04-17 01:26:34,884 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Exception raised. Traceback (most recent call last): File "awscli/s3transfer/tasks.py", line 139, in __call__ File "awscli/s3transfer/tasks.py", line 162, in _execute_main File "awscli/s3transfer/upload.py", line 798, in _main File "awscli/botocore/client.py", line 364, in _api_call File "awscli/botocore/client.py", line 733, in _make_api_call File "awscli/botocore/hooks.py", line 228, in emit File "awscli/botocore/hooks.py", line 211, in _emit File "awscli/customizations/s3errormsg.py", line 39, in enhance_error_msg File "awscli/customizations/s3errormsg.py", line 57, in _is_sigv4_error_message TypeError: argument of type 'NoneType' is not iterable 2026-04-17 01:26:34,886 - ThreadPoolExecutor-0_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None upload failed: ./hello.txt to s3://test-migration-src-102/hello.txt argument of type 'NoneType' is not iterable 2026-04-17 01:26:34,886 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread. --------------------------------------------- 2) rgw error log Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: ====== starting new request req=0x7f9567dde5a0 request_id=tx00000ee82cbfae0342e57-0069e18e83-301775-suwon-1 ===== Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s getting op 1 Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj verifying requester Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj normalizing buckets and tenants Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj init permissions Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj recalculating target Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj reading permissions Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj init op Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj verifying op mask Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj verifying op permissions Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=mw50, acct_name=mw50, subuser=, perm_mask=15, is_admin=0) mask=50 Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj Searching permissions for uid=mw50 Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj Found permission: 15 Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj Searching permissions for group=1 mask=50 Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj Permissions for group not found Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj Searching permissions for group=2 mask=50 Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj Permissions for group not found Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=mw50, acct_name=mw50, subuser=, perm_mask=15, is_admin=0), owner=mw50, perm=2 Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj verifying op params Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj pre-executing Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj check rate limiting Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.000000000s s3:put_obj executing Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.003999531s ERROR: failed to reserve notification on queue: :sdt-webhook-topic-1301. error: -22 Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.003999531s s3:put_obj completing Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.003999531s s3:put_obj op status=-22 Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: req 17186523406363012695 0.003999531s s3:put_obj http status=400 Apr 17 01:36:03 ljc-ceph-132 radosgw[3359289]: ====== req done req=0x7f9567dde5a0 op=put_obj bucket=test-migration-src-102 status=-22 http_status=400 latency=0.003999531s request_id=tx00000ee82cbfae0342e57-0069e18e83-301775-suwon-1 ====== --------------------------------------------- 3) Topic dump log & rados obj info root@ljc-ceph-132:/# radosgw-admin topic dump --topic sdt-webhook-topic-130 ERROR: failed to parse list entries from queue (skipping). error: (0) Success root@ljc-ceph-132:/# radosgw-admin topic dump --topic sdt-webhook-topic-1301 ERROR: failed to parse list entries from queue (skipping). error: (0) Success root@ljc-ceph-132:/# radosgw-admin topic dump --topic sdt-webhook-topic ERROR: failed to parse list entries from queue (skipping). error: (0) Success root@ljc-ceph-132:~# radosgw-admin topic get --topic sdt-webhook-topic-130 { "user": "mw50", "name": "sdt-webhook-topic-130", "dest": { "push_endpoint": "http://192.168.107.121:32167/webhook/v1/notifications", "push_endpoint_args": "", "push_endpoint_topic": "sdt-webhook-topic-130", "stored_secret": false, "persistent": true }, "arn": "arn:aws:sns:kr::sdt-webhook-topic-130", "opaqueData": "" } root@ljc-ceph-132:/# rados -p suwon-1.rgw.log ls | grep test-migration-src-102 pubsub..bucket.test-migration-src-102/a12c24ed-7a5a-4d7b-a065-c5e45dbb2216.108918.1 root@ljc-ceph-132:/# rados -p suwon-1.rgw.log getxattr pubsub..bucket.test-migration-src-102/a12c24ed-7a5a-4d7b-a065-c5e45dbb2216.108918.1 ceph.objclass.version $k8F/UNzFTUJSrJYe4Sv8jVwfroot@ljc-ceph-132:/# ---------------------------------------------------------------- Attached are the state and log details captured at the time of the issue. We also share the test results from another cluster with the same specifications after upgrading from reef to tentacle, covering each test case. ----------------------------------------------------------------------- Test Summary – Reef → Tentacle Upgrade & Bucket Notifications (Test #1) Procedure : Create a topic on Reef → set it as a bucket notification → upgrade to Tentacle → no topic‑attribute changes → perform an S3 PUT. Result : Notification delivered successfully. (Test #2) Procedure : Same as Test 1, but after the upgrade modify the topic’s attributes (e.g., endpoint, filter rules) before the S3 PUT. Result : Notification delivered successfully. (Test #3) Procedure : Upgrade from Reef to Tentacle first, then create a new topic on Tentacle, set it as a bucket notification, and perform an S3 PUT. Result : Notification delivered successfully. All three scenarios work as expected; upgrading the cluster and/or modifying topic attributes does not affect bucket‑notification functionality. Therefore, after the upgrade the topics and notifications appear to migrate and operate normally. However, during testing we encountered a situation similar to the current issue where the reserve queue fails. I would appreciate your thoughts on how to diagnose the root cause and what remedial actions can be taken.
participants (2)
-
steve jung
-
Yuval Lifshitz