radosgw-admin: the RGW management nightmare
Hi, We have a Ceph cluster hosting ~10 RGW realms and in our experience, radosgw-admin is making the Ceph admins life very hard... The opposite experience to other Ceph management tools. In addition to the lack of verifications before applying modifications that leads to the issues describe in another thread were you can have a configuration that cannot be commited and at the same time cannot be fixed, I have the feeling that one of the major issue is that radosgw-admin predates the rgw/zonegroup/zone paradigm and manages these 3 different kinds of objects as completely independent from the others. It is made worst by the fact that radosgw-admin defines a default object in each category, used implicitely (and slilently) if you don't select one specific object. The consequence of these 2 problems is that if you modify a zone, you can endup modifying the zonegroup of another realm than the one the zone belongs to (as there is no concept of a zone belonging to a zonegroup in radosgw-admin). It happens for example if you modify a zone (with zone modify) to make it the master zone (of a zonegroup in fact) and you don't specify the zonegroup the zone belongs to/is associated with: it will in fact update the default zonegroup that can be in another realm or at least not be the zonegroup the zone is associated with. A less impactful but nevertheless unexpected result is if you do something like: 'zonegroup list --rgw-realm my_realm'. You would expect to see only the zonegroups associated with the realm you specified where you'll get the list of all the zonegroups defined. I can imagine that chaning this is difficult without breaking backward compatibility. IMO, an improvement would be the ability not to have a default realm, zonegroup, zone. At least you would have an error if you don't specify explicitely the object to modify. Am I the only one struggling with radosgw-admin? Is there any plan to improve this? Best regards, Michel
On Mon, May 26, 2025 at 7:38 AM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote:
Hi,
We have a Ceph cluster hosting ~10 RGW realms and in our experience, radosgw-admin is making the Ceph admins life very hard... The opposite experience to other Ceph management tools.
In addition to the lack of verifications before applying modifications that leads to the issues describe in another thread were you can have a configuration that cannot be commited and at the same time cannot be fixed, I have the feeling that one of the major issue is that radosgw-admin predates the rgw/zonegroup/zone paradigm and manages these 3 different kinds of objects as completely independent from the others. It is made worst by the fact that radosgw-admin defines a default object in each category, used implicitely (and slilently) if you don't select one specific object. The consequence of these 2 problems is that if you modify a zone, you can endup modifying the zonegroup of another realm than the one the zone belongs to (as there is no concept of a zone belonging to a zonegroup in radosgw-admin). It happens for example if you modify a zone (with zone modify) to make it the master zone (of a zonegroup in fact) and you don't specify the zonegroup the zone belongs to/is associated with: it will in fact update the default zonegroup that can be in another realm or at least not be the zonegroup the zone is associated with. A less impactful but nevertheless unexpected result is if you do something like: 'zonegroup list --rgw-realm my_realm'. You would expect to see only the zonegroups associated with the realm you specified where you'll get the list of all the zonegroups defined.
I can imagine that chaning this is difficult without breaking backward compatibility. IMO, an improvement would be the ability not to have a default realm, zonegroup, zone. At least you would have an error if you don't specify explicitely the object to modify.
thanks Michel, this 'default' stuff was meant to be a convenience for clusters with a single zone each, but has indeed turned out to be a nightmare. prior to squid, it wasn't even possible to leave the 'default realm' unset. quoting the release note:
The radosgw-admin commands 'realm create' and 'realm pull' no longer set the default realm without --default.
there's a 'radosgw-admin realm default rm' command that can clear that default realm, if set. i don't know of a way to unset the default zone or zonegroup, though you at least had to opt in to those with --default
Am I the only one struggling with radosgw-admin? Is there any plan to improve this?
as you say, it's hard to make significant changes here without breaking existing deployments/automation. so there aren't any concrete plans, but we're open to ideas in the meantime, would you suggest any improvements to the documentation? for example, the note about 'default' behavior under https://docs.ceph.com/en/latest/radosgw/multisite/#create-a-realm
Best regards,
Michel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Casey, Thanks for you answer. Good news to know that in Squid it is possible to unset the default realm definition. In fact I was running Reef when I (or my colleagues) had all the last bad experiences but we upgraded to Squid yesterday! Nonetheless, I just add a look at `radosgw-admin --help` and the command `realm default rm` doesn't seems to be documented... Improvements to the documentation are probably the easiest and here are a few suggestions: - Warn that, despite there is normally a hierarchical relationship between realm, zonegroup and zone, radosgw-admin doesn't implement this hierarchy with the result that for a zone or zonegroup specifying a realm will not have any effect to restrict/select one (and the same with zonegroup and zone). - Mention that defining a default realm/zonegroup/zone is discouraged if you have several of thems for the reason above. This can lead to updating an object different from what you had in mind. - Remove the --default from suggested commands. When you are new you tend to add it, it you miss the note or didn't fully understand all the things it implies. As for small improvements to radosgw-admin that may avoid running into troubles, I could suggest: - Have a "default rm" command for zonegroup and zone too and document it in the help for all the objects that have it - Rewrite the help (may be not a small change in fact) so that you could have a hierarchical help giving more details than the summary on a given command (for example `realm zone modify --help` would tell you that if you use --master, this will modify the zonegroup the zone belongs to and that you have to mention it or the default zone will be used) - Do a minimum of validation when modifications are done to prevent have a configuration/period that cannot be commited anymore Really RGW are a very nice service in Ceph, easy to deploy in various scenarios when you use cephadm, but the radosgw-admin part of the configuration is very risky. This is not the case in other parts of Ceph and I think something should be done to put this part at the same level as other Ceph management commands/tools (I don't have in mind any where you can just by a small mistake break your configuration as there is enough validation done to prevent it and the famous --yes-i-really-want-it for the things that can make sense but are potentially dangerous). An issue has been open by Frederic after the discussion in another thread (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/6SD24SR7JBHF...) around this idea: https://tracker.ceph.com/issues/71406. Best regards, Michel Le 27/05/2025 à 17:53, Casey Bodley a écrit :
Hi,
We have a Ceph cluster hosting ~10 RGW realms and in our experience, radosgw-admin is making the Ceph admins life very hard... The opposite experience to other Ceph management tools.
In addition to the lack of verifications before applying modifications that leads to the issues describe in another thread were you can have a configuration that cannot be commited and at the same time cannot be fixed, I have the feeling that one of the major issue is that radosgw-admin predates the rgw/zonegroup/zone paradigm and manages these 3 different kinds of objects as completely independent from the others. It is made worst by the fact that radosgw-admin defines a default object in each category, used implicitely (and slilently) if you don't select one specific object. The consequence of these 2 problems is that if you modify a zone, you can endup modifying the zonegroup of another realm than the one the zone belongs to (as there is no concept of a zone belonging to a zonegroup in radosgw-admin). It happens for example if you modify a zone (with zone modify) to make it the master zone (of a zonegroup in fact) and you don't specify the zonegroup the zone belongs to/is associated with: it will in fact update the default zonegroup that can be in another realm or at least not be the zonegroup the zone is associated with. A less impactful but nevertheless unexpected result is if you do something like: 'zonegroup list --rgw-realm my_realm'. You would expect to see only the zonegroups associated with the realm you specified where you'll get the list of all the zonegroups defined.
I can imagine that chaning this is difficult without breaking backward compatibility. IMO, an improvement would be the ability not to have a default realm, zonegroup, zone. At least you would have an error if you don't specify explicitely the object to modify.
On Mon, May 26, 2025 at 7:38 AM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote: thanks Michel,
The radosgw-admin commands 'realm create' and 'realm pull' no longer set the default realm without --default.
this 'default' stuff was meant to be a convenience for clusters with a single zone each, but has indeed turned out to be a nightmare. prior to squid, it wasn't even possible to leave the 'default realm' unset. quoting the release note: there's a 'radosgw-admin realm default rm' command that can clear that default realm, if set. i don't know of a way to unset the default zone or zonegroup, though you at least had to opt in to those with --default
Am I the only one struggling with radosgw-admin? Is there any plan to improve this? as you say, it's hard to make significant changes here without breaking existing deployments/automation. so there aren't any concrete plans, but we're open to ideas
in the meantime, would you suggest any improvements to the documentation? for example, the note about 'default' behavior under https://docs.ceph.com/en/latest/radosgw/multisite/#create-a-realm
Best regards,
Michel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have had similar issues specifying what I'm actually trying to do. One idea I had, but it may be a bit much, would be to have an interactive radosgw-admin prompt. Similar to the way switch/router prompts work. e.g., you would enter a realm by selecting it, (the same way you enter an interface on a switch and then modify it). Then, modify that realm. Any modifications apply only to whatever realm you are in. And then, from there, enter a zonegroup, but the only ones available are the ones that exist in that realm. Same with zones, etc etc. Query commands can be performed from anywhere, but modifications must be done within the intended realm/zonegroup/zone. radosgw-admin> enable radosgw-admin> configure terminal radosgw-admin> realm my_realm radosgw-admin my_realm> no default radosgw-admin my_realm> zonegroup my_zonegroup radosgw-admin my_realm my_zonegroup> no default radosgw-admin my_realm my_zonegroup> zone my_zone radosgw-admin my_realm my_zonegroup my_zone> no default radosgw-admin my_realm my_zonegroup my_zone> exit radosgw-admin> rgw pending realm my_realm no default zonegroup my_zonegroup no default zone my_zone no default radosgw-admin> rgw commit Granted, this would require a lot of logic/work. And may be more restrictive than folks want/need. Just an idea. On 5/27/25 18:27, Michel Jouvin wrote:
Hi Casey,
Thanks for you answer. Good news to know that in Squid it is possible to unset the default realm definition. In fact I was running Reef when I (or my colleagues) had all the last bad experiences but we upgraded to Squid yesterday! Nonetheless, I just add a look at `radosgw-admin -- help` and the command `realm default rm` doesn't seems to be documented...
Improvements to the documentation are probably the easiest and here are a few suggestions:
- Warn that, despite there is normally a hierarchical relationship between realm, zonegroup and zone, radosgw-admin doesn't implement this hierarchy with the result that for a zone or zonegroup specifying a realm will not have any effect to restrict/select one (and the same with zonegroup and zone).
- Mention that defining a default realm/zonegroup/zone is discouraged if you have several of thems for the reason above. This can lead to updating an object different from what you had in mind.
- Remove the --default from suggested commands. When you are new you tend to add it, it you miss the note or didn't fully understand all the things it implies.
As for small improvements to radosgw-admin that may avoid running into troubles, I could suggest:
- Have a "default rm" command for zonegroup and zone too and document it in the help for all the objects that have it
- Rewrite the help (may be not a small change in fact) so that you could have a hierarchical help giving more details than the summary on a given command (for example `realm zone modify --help` would tell you that if you use --master, this will modify the zonegroup the zone belongs to and that you have to mention it or the default zone will be used)
- Do a minimum of validation when modifications are done to prevent have a configuration/period that cannot be commited anymore
Really RGW are a very nice service in Ceph, easy to deploy in various scenarios when you use cephadm, but the radosgw-admin part of the configuration is very risky. This is not the case in other parts of Ceph and I think something should be done to put this part at the same level as other Ceph management commands/tools (I don't have in mind any where you can just by a small mistake break your configuration as there is enough validation done to prevent it and the famous --yes-i-really-want- it for the things that can make sense but are potentially dangerous). An issue has been open by Frederic after the discussion in another thread (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/ thread/6SD24SR7JBHFZ4E5YTTTQ7NZULHYVHRA/) around this idea: https:// tracker.ceph.com/issues/71406.
Best regards,
Michel
Le 27/05/2025 à 17:53, Casey Bodley a écrit :
Hi,
We have a Ceph cluster hosting ~10 RGW realms and in our experience, radosgw-admin is making the Ceph admins life very hard... The opposite experience to other Ceph management tools.
In addition to the lack of verifications before applying modifications that leads to the issues describe in another thread were you can have a configuration that cannot be commited and at the same time cannot be fixed, I have the feeling that one of the major issue is that radosgw-admin predates the rgw/zonegroup/zone paradigm and manages these 3 different kinds of objects as completely independent from the others. It is made worst by the fact that radosgw-admin defines a default object in each category, used implicitely (and slilently) if you don't select one specific object. The consequence of these 2 problems is that if you modify a zone, you can endup modifying the zonegroup of another realm than the one the zone belongs to (as there is no concept of a zone belonging to a zonegroup in radosgw-admin). It happens for example if you modify a zone (with zone modify) to make it the master zone (of a zonegroup in fact) and you don't specify the zonegroup the zone belongs to/is associated with: it will in fact update the default zonegroup that can be in another realm or at least not be the zonegroup the zone is associated with. A less impactful but nevertheless unexpected result is if you do something like: 'zonegroup list --rgw-realm my_realm'. You would expect to see only the zonegroups associated with the realm you specified where you'll get the list of all the zonegroups defined.
I can imagine that chaning this is difficult without breaking backward compatibility. IMO, an improvement would be the ability not to have a default realm, zonegroup, zone. At least you would have an error if you don't specify explicitely the object to modify.
On Mon, May 26, 2025 at 7:38 AM Michel Jouvin <michel.jouvin@ijclab.in2p3.fr> wrote: thanks Michel,
The radosgw-admin commands 'realm create' and 'realm pull' no longer set the default realm without --default.
this 'default' stuff was meant to be a convenience for clusters with a single zone each, but has indeed turned out to be a nightmare. prior to squid, it wasn't even possible to leave the 'default realm' unset. quoting the release note: there's a 'radosgw-admin realm default rm' command that can clear that default realm, if set. i don't know of a way to unset the default zone or zonegroup, though you at least had to opt in to those with --default
Am I the only one struggling with radosgw-admin? Is there any plan to improve this? as you say, it's hard to make significant changes here without breaking existing deployments/automation. so there aren't any concrete plans, but we're open to ideas
in the meantime, would you suggest any improvements to the documentation? for example, the note about 'default' behavior under https://docs.ceph.com/en/latest/radosgw/multisite/#create-a-realm
Best regards,
Michel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Casey Bodley
-
Michel Jouvin
-
Robert Hish