Error Removing Zone from Zonegroup in Multisite Setup
Hi everyone, We have a Ceph multisite setup with a zonegroup containing three zones: one master and two secondary zones, one of which is designated as an archive. After recently upgrading to Ceph 19.2.1, I attempted to remove the archive zone from the zonegroup following the steps in the Ceph documentation. I ran radosgw-admin zonegroup remove --rgw-zonegroup=<ZONE_GROUP_NAME> --rgw-zone=<ZONE_NAME> --rgw-realm=<REALM_NAME> from the cluster containing the archive zone, followed by radosgw-admin period update --commit --rgw-zonegroup=<ZONE_GROUP_NAME> --rgw-zone=<ZONE_NAME> --rgw-realm=<REALM_NAME> However, the period update --commit command returns the error: ERROR: zonegroup <zonegroup_id> does not contain zone id <zone_id> The zone appears correctly in the zonegroup before the removal, and I expected the archive zone to be successfully removed with the period update committing without errors. Is there an additional step required, or could this be a bug related to Ceph 19.2.1? Any guidance or assistance would be appreciated. Thank you! Best regards, Mahnoosh
Hi, I've had similar issues before. radosgw-admin zonegroup/zone tooling is really bad in general. As far as I know you should remove zone before deleting it form zonegroup but I don't think I've ever managed to remove zone cleanly from the cluster. (I would be thankful if someone shared a correct way) (https://www.ibm.com/docs/en/storage-ceph/7?topic=zones-deleting-zone) To clean this up you may have to dump your zonegroup to file, edit json manually and upload it back to ceph without broken zone. If you have issues with deleting the zone itself from the local cluster RGW configuration you may have to hunt down objects containing it's IDs in .rgw.root pool. It's messy and dangerous so, don't forget to make a backup before you start ;) best regards, Adam W dniu 5.03.2025 o 18:02, Mahnoosh Shahidi pisze:
Hi everyone,
We have a Ceph multisite setup with a zonegroup containing three zones: one master and two secondary zones, one of which is designated as an archive. After recently upgrading to Ceph 19.2.1, I attempted to remove the archive zone from the zonegroup following the steps in the Ceph documentation.
I ran radosgw-admin zonegroup remove --rgw-zonegroup=<ZONE_GROUP_NAME> --rgw-zone=<ZONE_NAME> --rgw-realm=<REALM_NAME> from the cluster containing the archive zone, followed by radosgw-admin period update --commit --rgw-zonegroup=<ZONE_GROUP_NAME> --rgw-zone=<ZONE_NAME> --rgw-realm=<REALM_NAME>
However, the period update --commit command returns the error: ERROR: zonegroup <zonegroup_id> does not contain zone id <zone_id> The zone appears correctly in the zonegroup before the removal, and I expected the archive zone to be successfully removed with the period update committing without errors.
Is there an additional step required, or could this be a bug related to Ceph 19.2.1? Any guidance or assistance would be appreciated. Thank you!
Best regards, Mahnoosh _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Mar 5, 2025 at 9:34 AM Adam Prycki <aprycki@man.poznan.pl> wrote:
Hi,
I've had similar issues before. radosgw-admin zonegroup/zone tooling is really bad in general.
As far as I know you should remove zone before deleting it form zonegroup but I don't think I've ever managed to remove zone cleanly from the cluster. (I would be thankful if someone shared a correct way) (https://www.ibm.com/docs/en/storage-ceph/7?topic=zones-deleting-zone)
To clean this up you may have to dump your zonegroup to file, edit json manually and upload it back to ceph without broken zone.
If you have issues with deleting the zone itself from the local cluster RGW configuration you may have to hunt down objects containing it's IDs in .rgw.root pool. It's messy and dangerous so, don't forget to make a backup before you start ;)
best regards, Adam
W dniu 5.03.2025 o 18:02, Mahnoosh Shahidi pisze:
Hi everyone,
We have a Ceph multisite setup with a zonegroup containing three zones: one master and two secondary zones, one of which is designated as an archive. After recently upgrading to Ceph 19.2.1, I attempted to remove the archive zone from the zonegroup following the steps in the Ceph documentation.
I ran radosgw-admin zonegroup remove --rgw-zonegroup=<ZONE_GROUP_NAME> --rgw-zone=<ZONE_NAME> --rgw-realm=<REALM_NAME> from the cluster containing the archive zone, followed by radosgw-admin period update --commit --rgw-zonegroup=<ZONE_GROUP_NAME> --rgw-zone=<ZONE_NAME> --rgw-realm=<REALM_NAME>
However, the period update --commit command returns the error: ERROR: zonegroup <zonegroup_id> does not contain zone id <zone_id> The zone appears correctly in the zonegroup before the removal, and I expected the archive zone to be successfully removed with the period update committing without errors.
Is there an additional step required, or could this be a bug related to Ceph 19.2.1? Any guidance or assistance would be appreciated. Thank you!
it does appear to be a regression. have raised a tracker https://tracker.ceph.com/issues/70323 you don't see that zone entry when you run 'radosgw-admin zonegroup get' command on the archive zone anymore, do you? what happens when you run 'radosgw-admin period update' and then 'radosgw-admin period commit' separately?
I couldn't see the zone entry in the 'zonegroup get' response. Updating the period and commiting it separately worked fine. Thank you so much! Best regards, Mahnoosh On Wed, 5 Mar 2025, 19:35 Shilpa Manjrabad Jagannath, <smanjara@redhat.com> wrote:
On Wed, Mar 5, 2025 at 9:34 AM Adam Prycki <aprycki@man.poznan.pl> wrote:
Hi,
I've had similar issues before. radosgw-admin zonegroup/zone tooling is really bad in general.
As far as I know you should remove zone before deleting it form zonegroup but I don't think I've ever managed to remove zone cleanly from the cluster. (I would be thankful if someone shared a correct way) (https://www.ibm.com/docs/en/storage-ceph/7?topic=zones-deleting-zone)
To clean this up you may have to dump your zonegroup to file, edit json manually and upload it back to ceph without broken zone.
If you have issues with deleting the zone itself from the local cluster RGW configuration you may have to hunt down objects containing it's IDs in .rgw.root pool. It's messy and dangerous so, don't forget to make a backup before you start ;)
best regards, Adam
W dniu 5.03.2025 o 18:02, Mahnoosh Shahidi pisze:
Hi everyone,
We have a Ceph multisite setup with a zonegroup containing three zones: one master and two secondary zones, one of which is designated as an archive. After recently upgrading to Ceph 19.2.1, I attempted to remove the archive zone from the zonegroup following the steps in the Ceph documentation.
I ran radosgw-admin zonegroup remove --rgw-zonegroup=<ZONE_GROUP_NAME> --rgw-zone=<ZONE_NAME> --rgw-realm=<REALM_NAME> from the cluster containing the archive zone, followed by radosgw-admin period update --commit --rgw-zonegroup=<ZONE_GROUP_NAME> --rgw-zone=<ZONE_NAME> --rgw-realm=<REALM_NAME>
However, the period update --commit command returns the error: ERROR: zonegroup <zonegroup_id> does not contain zone id <zone_id> The zone appears correctly in the zonegroup before the removal, and I expected the archive zone to be successfully removed with the period update committing without errors.
Is there an additional step required, or could this be a bug related to Ceph 19.2.1? Any guidance or assistance would be appreciated. Thank you!
it does appear to be a regression. have raised a tracker https://tracker.ceph.com/issues/70323 you don't see that zone entry when you run 'radosgw-admin zonegroup get' command on the archive zone anymore, do you?
what happens when you run 'radosgw-admin period update' and then 'radosgw-admin period commit' separately? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Adam Prycki
-
Mahnoosh Shahidi
-
Shilpa Manjrabad Jagannath