missing amqp-exchange on bucket-notification with AMQP endpoint
Hello List, I'm trying to create a (S3-)bucket-notification into RabbitMQ via AMQP - on Ceph v15.2.1 octopus, using the official .deb packages on Debian Buster. I've created the following topic (directly via S3, not via pubsub REST API): <ListTopicsResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/"> <ListTopicsResult> <Topics> <member> <User>testuser</User> <Name>testtopic</Name> <EndPoint> <EndpointAddress>amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672</EndpointAddress> <EndpointArgs>Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=amqp.direct&push-endpoint=amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672</EndpointArgs> <EndpointTopic>testtopic</EndpointTopic> </EndPoint> <TopicArn>arn:aws:sns:de::testtopic</TopicArn> <OpaqueData></OpaqueData> </member> </Topics> </ListTopicsResult> ... </ListTopicsResponse> Then I've created the following bucket-notification <NotificationConfiguration> <TopicConfiguration> <Id>notify-psapp</Id> <Topic>arn:aws:sns:de::testtopic</Topic> <Event>s3:ObjectCreated:*</Event> <Event>s3:ObjectRemoved:*</Event> </TopicConfiguration> </NotificationConfiguration> When I upload a file into the bucket, the event itself seems to get fired, but radosgw keeps tell me that cmqp-exchange is not set 2020-04-20T12:24:29.935+0200 7ff01c5d3700 1 ====== starting new request req=0x7ff01c5cad50 ===== 2020-04-20T12:24:30.019+0200 7ff01c5d3700 1 ERROR: failed to create push endpoint: amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672 due to: pubsub endpoint configuration error: AMQP: missing amqp-exchange But it's there in the EndpointArgs, right? Or do I miss it somewhere else? Best Regards, Andreas
I've tried to debug this a bit.
<EndPoint> <EndpointAddress>amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672</EndpointAddress> <EndpointArgs>Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=amqp.direct&push-endpoint=amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672</EndpointArgs> <EndpointTopic>testtopic</EndpointTopic> </EndPoint>
For the above I was using the following request to create the topic - similar as it is described here [1]: https://ceph.example.com/?Action=CreateTopic&Name=testtopic&Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=amqp.direct&push-endpoint=amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672 (of course endpoint then URL-encoded) It seems to me that RGWHTTPArgs::parse() is not translating the "Attributes.entry.1..." strings into keys & values in its map. This are the keys & values that can now be found in the map: Found name: Attributes.entry.1.key Found value: amqp-exchange Found name: Attributes.entry.1.value Found value: amqp.direct Found name: push-endpoint Found value: amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672 If I simply change the request to: https://ceph.example.com/?Action=CreateTopic&Name=testtopic&amqp-exchange=amqp.direct&push-endpoint=amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672/foobar -> at voila, the entries in the map are correct Found name: amqp-exchange Found value: amqp.direct Found name: push-endpoint Found value: amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672 And then the bucket-notification works like it should. But I don't think the documentation is wrong, or is it? Cheers, Andreas [1] https://docs.ceph.com/docs/master/radosgw/notifications/#create-a-topic [2] Index: ceph-15.2.1/src/rgw/rgw_common.cc =================================================================== --- ceph-15.2.1.orig/src/rgw/rgw_common.cc +++ ceph-15.2.1/src/rgw/rgw_common.cc @@ -810,6 +810,8 @@ int RGWHTTPArgs::parse() string& name = nv.get_name(); string& val = nv.get_val(); + cout << "Found name: " << name << std::endl; + cout << "Found value: " << val << std::endl; append(name, val); }
Hi Andreas, The message format you tried to use is the standard one (the one being emitted from boto3, or any other AWS SDK [1]). It passes the arguments using 'x-www-form-urlencoded'. For example: POST / HTTP/1.1 Host: localhost:8000 Accept-Encoding: identity Date: Tue, 21 Apr 2020 08:52:35 GMT Content-Length: 293 Content-Type: application/x-www-form-urlencoded; charset=utf-8 Authorization: AWS KOC0EIWUFANCC3FX:8PunIZ4F36uK2c+3AKwhaKXgK84= User-Agent: Boto3/1.9.225 Python/2.7.17 Linux/5.5.13-200.fc31.x86_64 Botocore/1.15.28 Name=ajmmvc-1_topic_1& Attributes.entry.2.key=amqp-exchange& Attributes.entry.1.key=amqp-ack-level& Attributes.entry.2.value=amqp.direct& Version=2010-03-31& Attributes.entry.3.value=amqp%3A%2F%2F127.0.0.1%3A7001& Attributes.entry.1.value=none& Action=CreateTopic& Attributes.entry.3.key=push-endpoint Note that the arguments are passed inside the message body (no '?' in the URL), and are using the "Attributes" for all the non-standard parameters we added on top of the standard AWS topic creation command. The format that worked for you, is a non-standard one that we support, as documented for pubsub [2], which is using regular URL encoded parameters. Feel free to use either, but would recommend on the standard one. Anyway, thanks for pointing this confusion, will clarify that in the doc, and also fix the 'push-endpoint' part. Yuval [1] https://docs.aws.amazon.com/sns/latest/api/API_CreateTopic.html [2] https://docs.ceph.com/docs/master/radosgw/pubsub-module/#create-a-topic On Mon, Apr 20, 2020 at 8:05 PM Andreas Unterkircher <unki@netshadow.net> wrote:
I've tried to debug this a bit.
<EndPoint> <EndpointAddress>amqp://
rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672</EndpointAddress>
<EndpointArgs>Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=amqp.direct&push-endpoint=amqp:// rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672</EndpointArgs>
<EndpointTopic>testtopic</EndpointTopic> </EndPoint>
For the above I was using the following request to create the topic - similar as it is described here [1]:
(of course endpoint then URL-encoded)
It seems to me that RGWHTTPArgs::parse() is not translating the "Attributes.entry.1..." strings into keys & values in its map.
This are the keys & values that can now be found in the map:
Found name: Attributes.entry.1.key Found value: amqp-exchange Found name: Attributes.entry.1.value Found value: amqp.direct Found name: push-endpoint Found value: amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672
If I simply change the request to:
-> at voila, the entries in the map are correct
Found name: amqp-exchange Found value: amqp.direct Found name: push-endpoint Found value: amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672
And then the bucket-notification works like it should.
But I don't think the documentation is wrong, or is it?
Cheers, Andreas
[1] https://docs.ceph.com/docs/master/radosgw/notifications/#create-a-topic
[2] Index: ceph-15.2.1/src/rgw/rgw_common.cc =================================================================== --- ceph-15.2.1.orig/src/rgw/rgw_common.cc +++ ceph-15.2.1/src/rgw/rgw_common.cc @@ -810,6 +810,8 @@ int RGWHTTPArgs::parse() string& name = nv.get_name(); string& val = nv.get_val();
+ cout << "Found name: " << name << std::endl; + cout << "Found value: " << val << std::endl; append(name, val); } _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Dear Yuval!
The message format you tried to use is the standard one (the one being emitted from boto3, or any other AWS SDK [1]). It passes the arguments using 'x-www-form-urlencoded'. For example:
Thank you for your clarification! I've previously tried it as a x-www-form-urlencoded-body as well, but I have failed. That it was then working using the non-standard-parameters has lead me down the wrong road... But I have to admit that I'm still failing to create a topic the S3-way. I've tried it with curl, but as well with Postman. Even if I use your example-body, Ceph keeps telling me (at least) method-not-allowed. Is this maybe because I'm using an AWS Sig v4 to authenticate? This is the request I'm sending out: POST / HTTP/1.1 Content-Type: application/x-www-form-urlencoded; charset=utf-8 Accept-Encoding: identity Date: Tue, 23 Apr 2020 05:00:35 GMT X-Amz-Content-Sha256: e8d828552b412fde2cd686b0a984509bc485693a02e8c53ab84cf36d1dbb961a Host: s3.example.com X-Amz-Date: 2 as0200423T050035Z Authorization: AWS4-HMAC-SHA256 Credential=DNQXT3I8Z5MWDJ1A8YMP/20200423/de/s3/aws4_request, SignedHeaders=accept-encoding;content-type;date;host;x-amz-content-sha256;x-amz-date, Signature=fa65844ba997fe11e65be87a18f160afe1ea459892316d6060bbc663daf6eace User-Agent: PostmanRuntime/7.24.1 Accept: */* Connection: keep-alive Content-Length: 303 Name=ajmmvc-1_topic_1& Attributes.entry.2.key=amqp-exchange& Attributes.entry.1.key=amqp-ack-level& Attributes.entry.2.value=amqp.direct& Version=2010-03-31& Attributes.entry.3.value=amqp%3A%2F%2F127.0.0.1%3A7001& Attributes.entry.1.value=none& Action=CreateTopic& Attributes.entry.3.key=push-endpoint This is the response that comes back: HTTP/1.1 405 Method Not Allowed Content-Length: 200 x-amz-request-id: tx000000000000000000001-005ea12159-6e47a-s3-datacenter Accept-Ranges: bytes Content-Type: application/xml Date: Thu, 23 Apr 2020 05:02:17 GMT <?xml version="1.0" encoding="UTF-8"?><Error><Code>MethodNotAllowed</Code><RequestId>tx000000000000000000001-005ea12159-6e47a-s3-datacenter</RequestId><HostId>6e47a-s3-datacenter-de</HostId></Error> This is was radosgw is seeing at the same time 2020-04-23T07:02:17.745+0200 7f5aab2af700 20 final domain/bucket subdomain= domain=s3.example.com in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=s3.example.com s->info.request_uri=/ 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 meta>> HTTP_X_AMZ_DATE 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 x>> x-amz-content-sha256:e8d828552b412fde2cd686b0a984509bc485693a02e8c53ab84cf36d1dbb961a 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 x>> x-amz-date:20200423T050035Z 2020-04-23T07:02:17.745+0200 7f5aab2af700 20 req 1 0s get_handler handler=26RGWHandler_REST_Service_S3 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 handler=26RGWHandler_REST_Service_S3 2020-04-23T07:02:17.745+0200 7f5aab2af700 2 req 1 0s getting op 4 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 Content of POST: Name=ajmmvc-1_topic_1& Attributes.entry.2.key=amqp-exchange& Attributes.entry.1.key=amqp-ack-level& Attributes.entry.2.value=amqp.direct& Version=2010-03-31& Attributes.entry.3.value=amqp%3A%2F%2F127.0.0.1%3A7001& Attributes.entry.1.value=none& Action=CreateTopic& Attributes.entry.3.key=push-endpoint 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 Content of POST: Name=ajmmvc-1_topic_1& Attributes.entry.2.key=amqp-exchange& Attributes.entry.1.key=amqp-ack-level& Attributes.entry.2.value=amqp.direct& Version=2010-03-31& Attributes.entry.3.value=amqp%3A%2F%2F127.0.0.1%3A7001& Attributes.entry.1.value=none& Action=CreateTopic& Attributes.entry.3.key=push-endpoint 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 Content of POST: Name=ajmmvc-1_topic_1& Attributes.entry.2.key=amqp-exchange& Attributes.entry.1.key=amqp-ack-level& Attributes.entry.2.value=amqp.direct& Version=2010-03-31& Attributes.entry.3.value=amqp%3A%2F%2F127.0.0.1%3A7001& Attributes.entry.1.value=none& Action=CreateTopic& Attributes.entry.3.key=push-endpoint 2020-04-23T07:02:17.745+0200 7f5aab2af700 1 handler->ERRORHANDLER: err_no=-2003 new_err_no=-2003 2020-04-23T07:02:17.745+0200 7f5aab2af700 2 req 1 0s http status=405 2020-04-23T07:02:17.745+0200 7f5aab2af700 1 ====== req done req=0x7f5aab2a6d50 op status=0 http_status=405 latency=0s ====== Best Regards, Andreas
On Thu, Apr 23, 2020 at 8:28 AM Andreas Unterkircher <unki@netshadow.net> wrote:
Dear Yuval!
The message format you tried to use is the standard one (the one being emitted from boto3, or any other AWS SDK [1]). It passes the arguments using 'x-www-form-urlencoded'. For example:
Thank you for your clarification! I've previously tried it as a x-www-form-urlencoded-body as well, but I have failed. That it was then working using the non-standard-parameters has lead me down the wrong road... But I have to admit that I'm still failing to create a topic the S3-way.
I've tried it with curl, but as well with Postman. Even if I use your example-body, Ceph keeps telling me (at least) method-not-allowed.
Is this maybe because I'm using an AWS Sig v4 to authenticate?
yes, this is probably the issue. in the radosgw we use the same signature mechanism for S3 and for the other services (like topic creation). see this example: https://github.com/ceph/ceph/blob/master/examples/boto3/topic_with_endpoint.... (I guess we should also add that to the docs)
This is the request I'm sending out:
POST / HTTP/1.1 Content-Type: application/x-www-form-urlencoded; charset=utf-8 Accept-Encoding: identity Date: Tue, 23 Apr 2020 05:00:35 GMT X-Amz-Content-Sha256: e8d828552b412fde2cd686b0a984509bc485693a02e8c53ab84cf36d1dbb961a Host: s3.example.com X-Amz-Date: 2 as0200423T050035Z Authorization: AWS4-HMAC-SHA256 Credential=DNQXT3I8Z5MWDJ1A8YMP/20200423/de/s3/aws4_request, SignedHeaders=accept-encoding;content-type;date;host;x-amz-content-sha256;x-amz-date,
Signature=fa65844ba997fe11e65be87a18f160afe1ea459892316d6060bbc663daf6eace User-Agent: PostmanRuntime/7.24.1 Accept: */* Connection: keep-alive
Content-Length: 303
Name=ajmmvc-1_topic_1& Attributes.entry.2.key=amqp-exchange& Attributes.entry.1.key=amqp-ack-level& Attributes.entry.2.value=amqp.direct& Version=2010-03-31& Attributes.entry.3.value=amqp%3A%2F%2F127.0.0.1%3A7001& Attributes.entry.1.value=none& Action=CreateTopic& Attributes.entry.3.key=push-endpoint
This is the response that comes back:
HTTP/1.1 405 Method Not Allowed Content-Length: 200 x-amz-request-id: tx000000000000000000001-005ea12159-6e47a-s3-datacenter Accept-Ranges: bytes Content-Type: application/xml Date: Thu, 23 Apr 2020 05:02:17 GMT <?xml version="1.0"
encoding="UTF-8"?><Error><Code>MethodNotAllowed</Code><RequestId>tx000000000000000000001-005ea12159-6e47a-s3-datacenter</RequestId><HostId>6e47a-s3-datacenter-de</HostId></Error>
This is was radosgw is seeing at the same time
2020-04-23T07:02:17.745+0200 7f5aab2af700 20 final domain/bucket subdomain= domain=s3.example.com in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=s3.example.com s->info.request_uri=/ 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 meta>> HTTP_X_AMZ_DATE 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 x>>
x-amz-content-sha256:e8d828552b412fde2cd686b0a984509bc485693a02e8c53ab84cf36d1dbb961a 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 x>> x-amz-date:20200423T050035Z 2020-04-23T07:02:17.745+0200 7f5aab2af700 20 req 1 0s get_handler handler=26RGWHandler_REST_Service_S3 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 handler=26RGWHandler_REST_Service_S3 2020-04-23T07:02:17.745+0200 7f5aab2af700 2 req 1 0s getting op 4 2020-04-23T07:02:17.745+0200 7f5aab2af700 10 Content of POST: Name=ajmmvc-1_topic_1& Attributes.entry.2.key=amqp-exchange& Attributes.entry.1.key=amqp-ack-level& Attributes.entry.2.value=amqp.direct& Version=2010-03-31& Attributes.entry.3.value=amqp%3A%2F%2F127.0.0.1%3A7001& Attributes.entry.1.value=none& Action=CreateTopic& Attributes.entry.3.key=push-endpoint
2020-04-23T07:02:17.745+0200 7f5aab2af700 10 Content of POST: Name=ajmmvc-1_topic_1& Attributes.entry.2.key=amqp-exchange& Attributes.entry.1.key=amqp-ack-level& Attributes.entry.2.value=amqp.direct& Version=2010-03-31& Attributes.entry.3.value=amqp%3A%2F%2F127.0.0.1%3A7001& Attributes.entry.1.value=none& Action=CreateTopic& Attributes.entry.3.key=push-endpoint
2020-04-23T07:02:17.745+0200 7f5aab2af700 10 Content of POST: Name=ajmmvc-1_topic_1& Attributes.entry.2.key=amqp-exchange& Attributes.entry.1.key=amqp-ack-level& Attributes.entry.2.value=amqp.direct& Version=2010-03-31& Attributes.entry.3.value=amqp%3A%2F%2F127.0.0.1%3A7001& Attributes.entry.1.value=none& Action=CreateTopic& Attributes.entry.3.key=push-endpoint
2020-04-23T07:02:17.745+0200 7f5aab2af700 1 handler->ERRORHANDLER: err_no=-2003 new_err_no=-2003 2020-04-23T07:02:17.745+0200 7f5aab2af700 2 req 1 0s http status=405 2020-04-23T07:02:17.745+0200 7f5aab2af700 1 ====== req done req=0x7f5aab2a6d50 op status=0 http_status=405 latency=0s ======
Best Regards, Andreas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Andreas Unterkircher
-
Yuval Lifshitz