Fw: Incompatibilities (implicit_tenants & barbican) with Openstack after migrating from Ceph Luminous to Nautilus.

3 Mar 2020

      (resending to the new maillist)

Dear Casey, Dear All,

We tested the migration from Luminous to Nautilus and noticed two regressions breaking the RGW integration in Openstack:

1)  the following config parameter is not working on Nautilus but is valid on Luminous and on Master:
        rgw_keystone_implicit_tenants = swift

        In the log: parse error setting 'rgw_keystone_implicit_tenants' to 'swift' (Expected option value to be integer, got 'swift')

    This param is important to make RGW working for S3 and Swift.
    Setting it to false breaks swift/openstack and setting it to true makes S3 incompatible with dns-style bucketnames (with shared or public access).
    Please note that path-style bucketnames are deprecated by AWS and most clients are only supporting dns-style...

    Ref.:
        https://tracker.ceph.com/issues/24348
        https://github.com/ceph/ceph/commit/3ba7be8d1ac7ee43e69eebb58263cd080cca1d38

2) the server-side encryption (SSE-KMS) is broken on Nautilus:

    to reproduce the issue:
        s3cmd --access_key $ACCESSKEY --secret_key $SECRETKEY --host-bucket "%(bucket)s.$ENDPOINT" --host "$ENDPOINT" --region="$REGION" --signature-v2 --no-preserve --no-ssl --server-side-encryption --server-side-encryption-kms-id ${SECRET##*/} put helloenc.txt s3://testenc/

        output:
            upload: 'helloenc.txt' -> 's3://testenc/helloenc.txt'  [1 of 1]
            9 of 9   100% in    0s    37.50 B/s  done
            ERROR: S3 error: 403 (AccessDenied): Failed to retrieve the actual key, kms-keyid: cd0903db-c613-49be-96d9-165c02544bc7
        rgw log: see below

    TLDR: after investigating, I found that radosgw was actually getting the barbican secret correctly but the HTTP CODE (=200) validation was failing because of a bug in Nautilus.

    My understanding is following (please correct me):

        The bug in src/rgw/rgw_http_client.cc .

        Since Nautilus HTTP_CODE are converted into ERROR_CODE (200 becomes 0) in the request processing.
        This happens in RGWHTTPManager::reqs_thread_entry(), which centralizes the processing of (curl) HTTP Requests with multi-treading.

        This is fine but the member variable http_status of the class RGWHTTPClient is not updated with the resulting HTTP CODE, so the variable keeps its initial value of 0.

        Then in src/rgw/rgw_crypt.cc the logic is still verifying that http_status is in range [200,299] and this fails...

        I wrote the following oneliner bugfix for src/rgw/rgw_http_client.cc:

            diff --git a/src/rgw/rgw_http_client.cc b/src/rgw/rgw_http_client.cc
            index d0f0baead6..7c115293ad 100644
            --- a/src/rgw/rgw_http_client.cc
            +++ b/src/rgw/rgw_http_client.cc
            @@ -1146,6 +1146,7 @@ void *RGWHTTPManager::reqs_thread_entry()
                       status = -EAGAIN;
                     }
                     int id = req_data->id;
            +        req_data->client->http_status = http_status;
                    finish_request(req_data, status);
                     switch (result) {
                       case CURLE_OK:

        The s3cmd is then working fine with KMS server side encryption.

Questions:

  *       Could someone please write a fix for the regression of 1) and make a PR ?
  *       Could somebody also make a PR for 2?

Thank you for your help. :-)

Cheers
Francois Scheurer

rgw log:
        export CLUSTER=ceph; /home/local/ceph/build/bin/radosgw -f --cluster ${CLUSTER} --name client.rgw.$(hostname) --setuser ceph --setgroup ceph &
        tail -fn0 /var/log/ceph/ceph-client.rgw.ewos1-osd1-stage.log | less -IS
            2020-02-26 16:32:59.208 7fc1f1c54700 20 Getting KMS encryption key for key=cd0903db-c613-49be-96d9-165c02544bc7
            2020-02-26 16:32:59.208 7fc1f1c54700 20 Requesting secret from barbican url=http://keystone.service.stage.i.ewcs.ch:5000/v3/auth/tokens
            2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTPClient::process: http_status: 0
            2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTP::process
            2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTP::send
            2020-02-26 16:32:59.208 7fc1f1c54700 20 sending request to http://keystone.service.stage.i.ewcs.ch:5000/v3/auth/tokens
            2020-02-26 16:32:59.208 7fc1f1c54700 20 ssl verification is set to off
            2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTPManager::add_request: client->init_request(req_data): 0
            2020-02-26 16:32:59.208 7fc1f1c54700 20 register_request mgr=0x56374b865540 req_data->id=4, curl_handle=0x56374c77c4a0
            2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTPManager::signal_thread(): write(thread_pipe[1], (void *)&buf, sizeof(buf)): 4
            2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTPManager::add_request: signal_thread(): 0
            2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTP::send: rgw_http_manager->add_request(req): 0
            2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: RGWHTTP::process: send(req): 0
            2020-02-26 16:32:59.208 7fc1f1c54700 20 ewdebug: struct rgw_http_req_data : public RefCountedObject : int wait() : ret: 0
            2020-02-26 16:32:59.208 7fc2184a1700 20 link_request req_data=0x56374c96a240 req_data->id=4, curl_handle=0x56374c77c4a0
            2020-02-26 16:32:59.608 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: http_status: 201
            2020-02-26 16:32:59.608 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: rgw_http_error_to_errno(http_status): 0
            2020-02-26 16:32:59.608 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: finish_request(req_data, status): status: 0
            2020-02-26 16:32:59.608 7fc2184a1700 20 ewdebug: struct rgw_http_req_data : public RefCountedObject : void finish(int r) : ret: 0
            2020-02-26 16:32:59.652 7fc1f1c54700  5 ewdebug: request_key_from_barbican: Accept application/octet-stream X-Auth-Token gAAAAABeVo-xxx
            2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTPClient::process: http_status: 0
            2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTP::process
            2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTP::send
            2020-02-26 16:32:59.652 7fc1f1c54700 20 sending request to http://barbican.service.stage.i.ewcs.ch:9311/v1/secrets/cd0903db-c613-49be-9...
            2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTPManager::add_request: client->init_request(req_data): 0
            2020-02-26 16:32:59.652 7fc1f1c54700 20 register_request mgr=0x56374b865540 req_data->id=5, curl_handle=0x56374c77c4a0
            2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTPManager::signal_thread(): write(thread_pipe[1], (void *)&buf, sizeof(buf)): 4
            2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTPManager::add_request: signal_thread(): 0
            2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTP::send: rgw_http_manager->add_request(req): 0
            2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: RGWHTTP::process: send(req): 0
            2020-02-26 16:32:59.652 7fc1f1c54700 20 ewdebug: struct rgw_http_req_data : public RefCountedObject : int wait() : ret: 0
            2020-02-26 16:32:59.652 7fc2184a1700 20 link_request req_data=0x56374c96a240 req_data->id=5, curl_handle=0x56374c77c4a0
         => 2020-02-26 16:32:59.752 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: http_status: 200
            2020-02-26 16:32:59.752 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: rgw_http_error_to_errno(http_status): 0
            2020-02-26 16:32:59.752 7fc2184a1700 20 ewdebug: RGWHTTPManager::reqs_thread_entry: finish_request(req_data, status): status: 0
            2020-02-26 16:32:59.752 7fc2184a1700 20 ewdebug: struct rgw_http_req_data : public RefCountedObject : void finish(int r) : ret: 0
            2020-02-26 16:32:59.752 7fc1f1c54700  5 ewdebug: request_key_from_barbican: secret_req.process: 0
         => 2020-02-26 16:32:59.752 7fc1f1c54700  5 ewdebug: request_key_from_barbican: secret_req.get_http_status: 0
            2020-02-26 16:32:59.752 7fc1f1c54700  5 ewdebug: request_key_from_barbican: secret_req.get_http_status not in [200,299] range!
            2020-02-26 16:32:59.752 7fc1f1c54700  5 Failed to retrieve secret from barbican:cd0903db-c613-49be-96d9-165c02544bc7
            2020-02-26 16:32:59.752 7fc1f1c54700  5 ERROR: failed to retrieve actual key from key_id: cd0903db-c613-49be-96d9-165c02544bc7
            2020-02-26 16:32:59.752 7fc1f1c54700  2 req 1 1.092s s3:put_obj completing
            2020-02-26 16:32:59.752 7fc1f1c54700  2 req 1 1.092s s3:put_obj op status=-13
            2020-02-26 16:32:59.752 7fc1f1c54700  2 req 1 1.092s s3:put_obj http status=403
            2020-02-26 16:32:59.752 7fc1f1c54700  1 ====== req done req=0x56374c9808d0 op status=-13 http_status=403 latency=1.092s ======

        => we see that http_status is correct (200) but the variable secret_req.get_http_status (member of class RGWHTTPClient) is incorrect (0 instead of 200)

Scheurer François

Casey Bodley

Scheurer François

Scheurer François

Scheurer François

tags

participants (2)