Hello Ceph community, I am seeking advice regarding an RGW notification issue encountered after upgrading our cluster from Reef to Tentacle. We heavily use the RGW PubSub (topic/notification) feature. We recently upgraded to Tentacle specifically to leverage new reliability options such as time_to_live, max_retries, and retry_sleep_duration to prevent data loss during transmission. The Problem: After the upgrade, when using existing topics created in the Reef version, we started seeing the following error: "Failed to reserve notification on queue: error: -22" Consequently, all PUT requests to the affected buckets are failing with an error. (s3 response include error message "<Code>InvalidArgument</Code> Steps Taken (Troubleshooting): To resolve this without deleting our existing topics/notifications, we have attempted the following: - Restarted all RGW instances. - Redeployed the RGW gateways. - Performed rados clearomap on the pubsub..bucket.xxxx objects found in the logs. Despite these efforts, the PUT operations continue to fail with the same error. We are trying to find a fix that doesn't involve recreating the topics and notifications, as we are still in the process of debugging the root cause. Questions: - Has anyone encountered this -22 (EINVAL) error specifically during a Reef-to-Tentacle transition involving PubSub? - Are there any internal OMAP structure changes in Tentacle that might cause incompatibility with legacy Reef topics? - Any suggestions or ideas on how to clear this "queue reservation" error without losing the current notification configuration? Any insights or guidance would be greatly appreciated. Best regards,