ceph rgw zone create fails EINVAL
Hi, I'm running cephadm/reef 18.2.2. I'm trying to set up multisite. I created realm/zonegroup/master zone OK (I think!), edited the zonegroup json to include hostnames. I have this spec file for the secondary zone: rgw_zone: codfw rgw_realm_token: "SECRET" placement: label: "rgw" [I get "SECRET" by doing ceph rgw realm tokens on the master, and C&Ping the field labelled "token"] If I then try and apply this with: ceph rgw zone create -i /root/rgw_secondary.yaml It doesn't work, and I get an unhelpful backtrace: Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call return self.func(mgr, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/rgw/module.py", line 96, in wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/rgw/module.py", line 304, in _cmd_rgw_zone_create return HandleCommandResult(retval=0, stdout=f"Zones {', '.join(created_zones)} created successfully") ^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: sequence item 0: expected str instance, int found I assume I've messed up the spec file, but it looks like the one in the docs[0]. Can anyone point me in the right direction, please? [if the underlying command emits anything useful, I can't find it in the logs] Thanks, Matthew [0] https://docs.ceph.com/en/reef/mgr/rgw/#realm-credentials-token
I think this is at least partially a code bug in the rgw module. Where it's actually failing in the traceback is generating the return message for the user at the end, because it assumes `created_zones` will always be a list of strings and that seems to not be the case in any error scenario. That caused that error to be the one printed instead of the actual underlying failure. From what I can see looking at the code, 1 of 3 error conditions happened. 1) It may have actually failed parsing the spec. You may be able to test this by using `orch apply` instead with the --dry-run flag so it doesn't actually make daemons but should still have to parse the spec. 2) If either the inbuf (-i <filepath>) or zone_name and realm_token aren't provided it errors out. That one shouldn't be the case since you used -i in your command. 3) It actually failed creating the zone. Unfortunately, there seems to be a whole bunch of reasons it could fail doing so, as it does much more than just creating the zone. From what I can see, this can fail if there is no realm token provided, there is no zone provided, the zone already exists, the token contains no endpoint, secreate, access key or realm name, it fails to pull the realm (radosgw-admin realm pull --url <realm_token enpoint> --access-key <access key> --secret <realm token secret>), it fail to find the master zonegroup (using a `radosgw-admin zonegroup get...` command with the realm from the spec passed in), it fails actually creating the zone (using a `radosgw-admin zone create` command with the --master, --access-key, --secret and --endpoints params all filled in), or it fails updating the period after creating the zone. I wish I knew this module a bit better so I could provide something more useful than a massive list of potential failure causes, but unfortunately I do not. I am at least going to create a patch to fix the error handling issue here though. On Wed, Jun 19, 2024 at 2:16 PM Matthew Vernon <mvernon@wikimedia.org> wrote:
Hi,
I'm running cephadm/reef 18.2.2. I'm trying to set up multisite.
I created realm/zonegroup/master zone OK (I think!), edited the zonegroup json to include hostnames. I have this spec file for the secondary zone:
rgw_zone: codfw rgw_realm_token: "SECRET" placement: label: "rgw"
[I get "SECRET" by doing ceph rgw realm tokens on the master, and C&Ping the field labelled "token"]
If I then try and apply this with: ceph rgw zone create -i /root/rgw_secondary.yaml
It doesn't work, and I get an unhelpful backtrace: Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1811, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/mgr_module.py", line 474, in call return self.func(mgr, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/rgw/module.py", line 96, in wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/ceph/mgr/rgw/module.py", line 304, in _cmd_rgw_zone_create return HandleCommandResult(retval=0, stdout=f"Zones {', '.join(created_zones)} created successfully")
^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: sequence item 0: expected str instance, int found
I assume I've messed up the spec file, but it looks like the one in the docs[0]. Can anyone point me in the right direction, please?
[if the underlying command emits anything useful, I can't find it in the logs]
Thanks,
Matthew
[0] https://docs.ceph.com/en/reef/mgr/rgw/#realm-credentials-token _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 19/06/2024 19:45, Adam King wrote:
I think this is at least partially a code bug in the rgw module. Where
...the code path seems to have a bunch of places it might raise an exception; are those likely to result in some entry in a log-file? I've not found anything, which is making working out what the problem is quite challenging... Thanks, Matthew
On 24/06/2024 20:49, Matthew Vernon wrote:
On 19/06/2024 19:45, Adam King wrote:
I think this is at least partially a code bug in the rgw module. Where
...the code path seems to have a bunch of places it might raise an exception; are those likely to result in some entry in a log-file? I've
Ah, I do now find: 2024-06-24T17:33:26.880065+00:00 moss-be2001 ceph-mgr[129346]: [rgw ERROR root] Non-zero return from ['radosgw-admin', '-k', '/var/lib/ceph/mgr/ceph-moss-be2001.qvwcaq/keyring', '-n', 'mgr.moss-be2001.qvwcaq', 'realm', 'pull', '--url', 'https://apus.svc.eqiad.wmnet:443', '--access-key', 'REDACTED', '--secret', 'REDACTED', '--rgw-realm', 'apus']: request failed: (5) Input/output error EIO is an odd sort of error [doesn't sound very network-y], and I don't think I see any corresponding request in the radosgw logs in the primary zone. From the CLI outside the container I can do e.g. curl https://apus.svc.eqiad.wmnet/ just fine, are there other things worth checking here? Could it matter that the mgr node isn't an rgw? Thanks, Matthew
On 24/06/2024 21:18, Matthew Vernon wrote:
2024-06-24T17:33:26.880065+00:00 moss-be2001 ceph-mgr[129346]: [rgw ERROR root] Non-zero return from ['radosgw-admin', '-k', '/var/lib/ceph/mgr/ceph-moss-be2001.qvwcaq/keyring', '-n', 'mgr.moss-be2001.qvwcaq', 'realm', 'pull', '--url', 'https://apus.svc.eqiad.wmnet:443', '--access-key', 'REDACTED', '--secret', 'REDACTED', '--rgw-realm', 'apus']: request failed: (5) Input/output error
EIO is an odd sort of error [doesn't sound very network-y], and I don't think I see any corresponding request in the radosgw logs in the primary zone. From the CLI outside the container I can do e.g. curl https://apus.svc.eqiad.wmnet/ just fine, are there other things worth checking here? Could it matter that the mgr node isn't an rgw?
...the answer turned out to be "container image lacked the relevant CA details to validate the TLS of the other end". Regards, Matthew
On 6/25/24 3:21 PM, Matthew Vernon wrote:
On 24/06/2024 21:18, Matthew Vernon wrote:
2024-06-24T17:33:26.880065+00:00 moss-be2001 ceph-mgr[129346]: [rgw ERROR root] Non-zero return from ['radosgw-admin', '-k', '/var/lib/ceph/mgr/ceph-moss-be2001.qvwcaq/keyring', '-n', 'mgr.moss-be2001.qvwcaq', 'realm', 'pull', '--url', 'https://apus.svc.eqiad.wmnet:443', '--access-key', 'REDACTED', '--secret', 'REDACTED', '--rgw-realm', 'apus']: request failed: (5) Input/output error
EIO is an odd sort of error [doesn't sound very network-y], and I don't think I see any corresponding request in the radosgw logs in the primary zone. From the CLI outside the container I can do e.g. curl https://apus.svc.eqiad.wmnet/ just fine, are there other things worth checking here? Could it matter that the mgr node isn't an rgw?
...the answer turned out to be "container image lacked the relevant CA details to validate the TLS of the other end".
Also, for the record, radosgw-admin logs do not end up in the same log file as RGW's logs. Each invocation of radosgw-admin makes it's own log file for the run of that command. (This is because radosgw-admin is really a stripped down version of RGW itself, and it does not communicate with the running RGWs, but connects to the Ceph cluster directly.) They're generally small, and frequently empty, but should have error messages in them on failure. Daniel
Interesting. Given this is coming from a radosgw-admin call being done from within the rgw mgr module, I wonder if a radosgw-admin log file is ending up in the active mgr container when this happens. On Wed, Jun 26, 2024 at 9:04 AM Daniel Gryniewicz <dang@redhat.com> wrote:
On 6/25/24 3:21 PM, Matthew Vernon wrote:
On 24/06/2024 21:18, Matthew Vernon wrote:
2024-06-24T17:33:26.880065+00:00 moss-be2001 ceph-mgr[129346]: [rgw ERROR root] Non-zero return from ['radosgw-admin', '-k', '/var/lib/ceph/mgr/ceph-moss-be2001.qvwcaq/keyring', '-n', 'mgr.moss-be2001.qvwcaq', 'realm', 'pull', '--url', 'https://apus.svc.eqiad.wmnet:443', '--access-key', 'REDACTED', '--secret', 'REDACTED', '--rgw-realm', 'apus']: request failed: (5) Input/output error
EIO is an odd sort of error [doesn't sound very network-y], and I don't think I see any corresponding request in the radosgw logs in the primary zone. From the CLI outside the container I can do e.g. curl https://apus.svc.eqiad.wmnet/ just fine, are there other things worth checking here? Could it matter that the mgr node isn't an rgw?
...the answer turned out to be "container image lacked the relevant CA details to validate the TLS of the other end".
Also, for the record, radosgw-admin logs do not end up in the same log file as RGW's logs. Each invocation of radosgw-admin makes it's own log file for the run of that command. (This is because radosgw-admin is really a stripped down version of RGW itself, and it does not communicate with the running RGWs, but connects to the Ceph cluster directly.) They're generally small, and frequently empty, but should have error messages in them on failure.
Daniel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I would guess that it probably does, but I don't know for sure. Daniel On 6/26/24 10:04 AM, Adam King wrote:
Interesting. Given this is coming from a radosgw-admin call being done from within the rgw mgr module, I wonder if a radosgw-admin log file is ending up in the active mgr container when this happens.
On Wed, Jun 26, 2024 at 9:04 AM Daniel Gryniewicz <dang@redhat.com <mailto:dang@redhat.com>> wrote:
On 6/25/24 3:21 PM, Matthew Vernon wrote: > On 24/06/2024 21:18, Matthew Vernon wrote: > >> 2024-06-24T17:33:26.880065+00:00 moss-be2001 ceph-mgr[129346]: [rgw >> ERROR root] Non-zero return from ['radosgw-admin', '-k', >> '/var/lib/ceph/mgr/ceph-moss-be2001.qvwcaq/keyring', '-n', >> 'mgr.moss-be2001.qvwcaq', 'realm', 'pull', '--url', >> 'https://apus.svc.eqiad.wmnet:443 <https://apus.svc.eqiad.wmnet:443>', '--access-key', 'REDACTED', >> '--secret', 'REDACTED', '--rgw-realm', 'apus']: request failed: (5) >> Input/output error >> >> EIO is an odd sort of error [doesn't sound very network-y], and I >> don't think I see any corresponding request in the radosgw logs in the >> primary zone. From the CLI outside the container I can do e.g. curl >> https://apus.svc.eqiad.wmnet/ <https://apus.svc.eqiad.wmnet/> just fine, are there other things worth >> checking here? Could it matter that the mgr node isn't an rgw? > > ...the answer turned out to be "container image lacked the relevant CA > details to validate the TLS of the other end". >
Also, for the record, radosgw-admin logs do not end up in the same log file as RGW's logs. Each invocation of radosgw-admin makes it's own log file for the run of that command. (This is because radosgw-admin is really a stripped down version of RGW itself, and it does not communicate with the running RGWs, but connects to the Ceph cluster directly.) They're generally small, and frequently empty, but should have error messages in them on failure.
Daniel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
participants (3)
-
Adam King
-
Daniel Gryniewicz
-
Matthew Vernon