Interesting. Given this is coming from a radosgw-admin call being done from within the rgw mgr module, I wonder if a radosgw-admin log file is ending up in the active mgr container when this happens. On Wed, Jun 26, 2024 at 9:04 AM Daniel Gryniewicz <dang@redhat.com> wrote:
On 6/25/24 3:21 PM, Matthew Vernon wrote:
On 24/06/2024 21:18, Matthew Vernon wrote:
2024-06-24T17:33:26.880065+00:00 moss-be2001 ceph-mgr[129346]: [rgw ERROR root] Non-zero return from ['radosgw-admin', '-k', '/var/lib/ceph/mgr/ceph-moss-be2001.qvwcaq/keyring', '-n', 'mgr.moss-be2001.qvwcaq', 'realm', 'pull', '--url', 'https://apus.svc.eqiad.wmnet:443', '--access-key', 'REDACTED', '--secret', 'REDACTED', '--rgw-realm', 'apus']: request failed: (5) Input/output error
EIO is an odd sort of error [doesn't sound very network-y], and I don't think I see any corresponding request in the radosgw logs in the primary zone. From the CLI outside the container I can do e.g. curl https://apus.svc.eqiad.wmnet/ just fine, are there other things worth checking here? Could it matter that the mgr node isn't an rgw?
...the answer turned out to be "container image lacked the relevant CA details to validate the TLS of the other end".
Also, for the record, radosgw-admin logs do not end up in the same log file as RGW's logs. Each invocation of radosgw-admin makes it's own log file for the run of that command. (This is because radosgw-admin is really a stripped down version of RGW itself, and it does not communicate with the running RGWs, but connects to the Ceph cluster directly.) They're generally small, and frequently empty, but should have error messages in them on failure.
Daniel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io