Hi Steve,

There were several internal format changes between reef and squid. There is a migration process that should kick in once you enable the notification_v2 zone feature. However, everything should still work even if that feature is not enabled.

Could you please share a snippet of the rgw debug logs showing more detailed error information?

As a last resort, you can use the new "topic dump" command to get all of the notifications in json format (so the information is not lost), and then delete and recreate the topics.

Yuval

On Tue, Apr 14, 2026 at 8:42 PM steve jung via dev <dev@ceph.io> wrote:

Hello Ceph community,

I am seeking advice regarding an RGW notification issue encountered after upgrading our cluster from Reef to Tentacle.

We heavily use the RGW PubSub (topic/notification) feature. We recently upgraded to Tentacle specifically to leverage new reliability options such as time_to_live, max_retries, and retry_sleep_duration to prevent data loss during transmission.

The Problem:

After the upgrade, when using existing topics created in the Reef version, we started seeing the following error:
"Failed to reserve notification on queue: error: -22"
Consequently, all PUT requests to the affected buckets are failing with an error.
(s3 response include error message "<Code>InvalidArgument</Code>
Steps Taken (Troubleshooting):

To resolve this without deleting our existing topics/notifications, we have attempted the following:
- Restarted all RGW instances.
- Redeployed the RGW gateways.
- Performed rados clearomap on the pubsub..bucket.xxxx objects found in the logs.
Despite these efforts, the PUT operations continue to fail with the same error. We are trying to find a fix that doesn't involve recreating the topics and notifications, as we are still in the process of debugging the root cause.

Questions:

- Has anyone encountered this -22 (EINVAL) error specifically during a Reef-to-Tentacle transition involving PubSub?
- Are there any internal OMAP structure changes in Tentacle that might cause incompatibility with legacy Reef topics?
- Any suggestions or ideas on how to clear this "queue reservation" error without losing the current notification configuration?

Any insights or guidance would be greatly appreciated.

Best regards,