I solved this. It has multiple layers. 1. RGW_API_HOST is no longer available in 17.2.6 as a configuration option for the ceph mgr. (I was wrong below when I said it could be queried on an *upgraded* host with: # ceph dashboard get-rgw-api-host You *can* query it with: # ceph config dump | grep mgr | grep dashboard But because the mgr doesn't use it, it's existence in the config doesn't matter. My setup uses a DNS record that points to multiple IP addresses for the rgw servers. This record is s3.my.dom, and these IP numbers are VIPs that are controlled by keepalived, to share them amongst the radosgw servers in the case one of the radosgw servers dies, one of the others will answer for it. But each radosgw server itself has a name similar to: rgw1.my.dom, rgw2.my.dom, rgw3.my.dom, in DNS each of these points to the IP of the actual server, (not the VIPs above). In 17.2.6, I *can* do this: $ aws --endpoint https://s3.my.dom s3 ls s3://mybucket But, in 17.2.6, I can no longer do: $ aws --endpoint https://rgw1.my.dom s3 ls s3://mybucket It returns a "NoSuchBucket" error with a 403 error. This looks to be the same error that the ceph mgr is returning in the GUI, as it tries to query the individual radosgw server name, NOT the s3.my.dom which it was before the upgrade to 17.2.6 (It was using the RGW_API_HOST which I have set to s3.my.dom. In the radosgw zonegroup, (radosgw-admin zonegroup get) I have: "endpoints": [     "https://s3.my.dom"] "hostnames": [   "s3.my.dom"] and in the individual zone definition I have: "endpoints": [   "https://s3.my.dom"] The code in rgw_rest.cc has this section:     if (subdomain.empty()         && (domain.empty() || domain != info.host)         && !looks_like_ip_address(info.host.c_str())         && RGWHandler_REST::validate_bucket_name(info.host) == 0         && !(hostnames_set.empty() && hostnames_s3website_set.empty())) {       subdomain.append(info.host);       in_hosted_domain = 1;     } and later this:     if (in_hosted_domain && !subdomain.empty()) {       string encoded_bucket = "/";       encoded_bucket.append(subdomain);       if (s->info.request_uri[0] != '/')         encoded_bucket.append("/");       encoded_bucket.append(s->info.request_uri);       s->info.request_uri = encoded_bucket;     } In my situation, this ends up with the bucket name being changed, resulting in the NoSuchBucket error. The way to fix this is to have "rgw dns name" set separately for each rgw host in ceph.conf and restarting the radosgw servers. That way, the rgw hostname is set and the bucket name does not get changed. It appears that in my setup, without "rgw dns name" set,along with the other settings I have, radosgw ASSUMES? we are using the newer domain style bucket names (we are not), and thatthe radosgw servers no longer recognize (with my settings) their own server name as valid. Adding "rgw dns name" for each server to ceph.confallows the aws cli to work with --endpoint set to the individual server name and for the ceph-mgr to work when querying radosgw information. In pacific 16.x.y this was not a problem. -Chris On Wednesday, August 30, 2023 at 05:08:36 AM MDT, Eugen Block <eblock@nde.ag> wrote: Hi, there have been multiple discussions on this list without any  satisfying solution for all possible configurations. One of the  changes [1] made in Pacific was to use hostname instead of IP, but it  only uses the shortname (you can check the "hostname" in 'ceph service  dump' output. But this seems to only impact the dashboard access if  you have ssl-verify set to true. I'm still waiting for a solution as  well for a customer cluster which uses wildcard certificates only,  until then we let ssl-verify disabled. But I didn't check the tracker  for any pending tickets, so someone might be working on it. Regards, Eugen [1] https://github.com/ceph/ceph/pull/47207/files Zitat von Christopher Durham <caduceus42@aol.com>:

Hi, I am using 17.2.6 on Rocky Linux 8 The ceph mgr dashboard, in my situation, (bare metal install,  upgraded from 15->16-> 17.2.6), can no longer hit the  ObjectStore->(Daemons,Users,Buckets) pages.

When I try to hit those pages, it gives an error: RGW REST API failed request with status code 403 {"Code":  "AccessDenied", RequestId: "xxxxxxx", HostId: "yyyy-<my zone>"}

The log of the rgw server it hit has:

"GET /admin/metadata/user?myself HTTP/1.1" 403 125

It appears that the mgr dashboard setting RGW_API_HOST is no longer  an option that can be set, nor does that name exist anywhere under  /usr/share/ceph/mgr/dashboard, and:

# ceph dashboard set-rgw-api-host <host>

is no longer in existence in 17.2.6

However, since my situation is an upgrade, the config value still  exists in my config, and I can retrieve it with:

# ceph dashboard get-rgw-api-host

To get the  to work in my situation, I have modified  /usr/share/ceph/mgr/dashboard/settings.py and re-added RGW_API_HOST  to the Options class using

RGW_API_HOST = Settings('', [dict,str])

I then modified  /usr/share/ceph/mgr/dashboard/services/rgw_request.py such that each  rgw daemon retrieved has its 'host' member set to  Settings.RGW_API_HOST.

Then after restarting the mgr, I was able to access the  Objectstore->(Daemons,Users,Buckets) pages in the dashboard.

HOWEVER, I know this is NOT the right way to fix this, it is a hack.  It seems like the dashboard is trying to contact an rgw server  individually. For us, the RGW_API_HOST is a name in DNS: s3.my.dom, that has multiple A records, one for each  of our rgw servers, each of which have the *same* SSL cert with CN  and SubjectAltNames that allow the cert to present itself as both s3.my.dom as well as the  individual host name (SubjectAltName has ALL the rgw servers in it).  This works well for us and has done so since 15.x.y, The endpoint for the zone is set to s3.my.dom.  Thus my users only have a single endpoint to care about, unless  there is a failure situation onan rgw server. (We have other ways of  handling that). Any thoughts on the CORRECT way to handle this so I can have the  ceph dashboard work with the ObjectStore->(Daemons,Users,Buckets)  pages? Thanks. -Chris

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io