(cc Yuval and Ali) the problem is that v1 topics did not support multisite, so i'm guessing you had to recreate those same topics on each zone. the v2 migration code can only run on the primary zone, so multisite can't automatically reconcile those v1 topics on your secondary zone
WARNING: topic migration in process. please try again later
you're seeing this warning on the secondary zone because the v1 topics still exist there. that's also why "radosgw-admin topic list" is only showing the v1 topics all v1 topics were stored in a single rados object per tenant, named "pubsub.{tenant}" in the site2.rgw.log pool. if you're not using tenants, that would just be "pubsub." since the v2 topic metadata has already replicated, it should be safe to delete any of those v1 rados objects manually to complete that v2 migration: 1. first make a local backup: $ rados -p site2.rgw.log get pubsub. pubsub.bak 2. delete the rados object: $ rados -p site2.rgw.log rm pubsub. 3. verify that v2 topics are now visible to radosgw-admin: $ radosgw-admin topic list 4. restart radosgw instances that are still caching the v1 topic metadata repeat steps 1 and 2 for other tenant names as needed. if necessary, you can restore the v1 metadata with: $ rados -p site2.rgw.log put pubsub. pubsub.bak On Thu, Dec 4, 2025 at 6:08 AM Lázár György via ceph-users <ceph-users@ceph.io> wrote:
Hello, I have an object multisite setup. I'm on squid 19.2.3 (upgraded, not a greenfield deployment - both clusters are on the same builds), so I upgraded to notification_v2 as well. However, while the primary zone converted the notifications and topics to v2 easily, the secondary zone seems to have issues with this (topics and notifications are not synced to the secondary, but the rados objects are synced, shown via rados -p site2.rgw.log -N notif ls. The radosgw-admin topic list command also gives back the v1 format of the json output on the secondary zone, while having the v2 json output on the primary). What's (almost) certain is, that one of the clusters (the primary) wants to use v2, and the other the v1 notification style, and mixing these two causes issues (topics randomly breaking, etc). Also, since 19.2.3, in theory, if I create a topic on the secondary zone, the operation should be forwarded to the primary zone. While this happens, the topic creation still fails with an 503 error, while the topic still gets created. The rgw on the secondary site after this half-failed operation also spits out logs like: rgw debug 2025-11-26T11:51:14.241+0000 7ffa54948640 1 req 12442569609481506669 0.000000000s sns:pubsub_topic_create ERROR: topic not found rgw debug 2025-11-26T11:51:14.254+0000 7ffa2e8fc640 1 req 12442569609481506669 0.013000983s sns:pubsub_topic_create WARNING: topic migration in process. please try again later I tried to create another multisite setup (same clusters, different realm), and there the notification v2 scheme works without an issue. I think if I would delete the secondary zone and re-create it (including the pools), it would fix the issue, but since that operation has a relatively large blast radius (especially if something goes wrong, but there are some services that use only the secondary zone, so they should be redirected beforehand), I'm a bit hesitant to do it (before the christmas holidays). I also tried searching for these error messages on google, with no luck. I also tried searching for these error messages in the RGW source code, and still no luck. I can find them in the compiled RGW binary at least.
Any help would be greatly appreciated! I would be also very happy if I wouldn't have to nuke the secondary zone.
Redards, György Lázár _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io