RGW topic created in wrong (default) tenant
Hi I'm using Ceph Pacific 16.2.1 I'm creating a topic as a user which belongs to a non-default tenant. I'm using AWS CLI 2 with v3 authentication enabled aws --profile=ceph-myprofile --endpoint=$HOST_S3_API --region="" sns create-topic --name=fishtopic --attributes='{"push-endpoint": " http://my-ceph-source-svc.default.svc.cluster.local"}' { "TopicArn": "arn:aws:sns:default::fishtopic" } topic is created in default tenant though. User can list topics but see topics from the default tenant. aws --profile=ceph-myprofile --endpoint=$HOST_S3_API --region="" sns list-topics { "Topics": [ { "TopicArn": "arn:aws:sns:default::fishtopic" } ] } Topic is in default tenant # radosgw-admin topic list --uid none { "topics": [ { "topic": { "user": "", "name": "fishtopic", "dest": { "bucket_name": "", "oid_prefix": "", "push_endpoint": " http://my-ceph-source-svc.default.svc.cluster.local", "push_endpoint_args": "Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value= http://my-ceph-source-svc.default.svc.cluster.local &Version=2010-03-31&push-endpoint= http://my-ceph-source-svc.default.svc.cluster.local", "push_endpoint_topic": "fishtopic", "stored_secret": "false", "persistent": "false" }, "arn": "arn:aws:sns:default::fishtopic", "opaqueData": "" }, "subs": [] } ] } When I create a topic over HTTP with a federated user, the topic is created in the correct (user's) tenant. For some reason the "user" below is "marvel", which is actually the name of the tenant. Possibly the topic is not owned by the user but rather the tenant. radosgw-admin topic list --tenant marvel --uid none { "topics": [ { "topic": { "user": "marvel", "name": "MyTopic", "dest": { "bucket_name": "", "oid_prefix": "", "push_endpoint": "amqp://127.0.0.1", "push_endpoint_args": "amqp-exchange=rgw-exchange&push-endpoint=amqp://127.0.0.1 &use-ssl=false&verify-ssl=false", "push_endpoint_topic": "MyTopic", "stored_secret": "false", "persistent": "false" }, "arn": "arn:aws:sns:default:marvel:MyTopic", "opaqueData": "" }, "subs": [] } ] } Also, what permissions are checked when creating a topic? It seems so far I can create a topic without granting any special permissions? Regards Daniel
On Wed, Jun 23, 2021 at 2:21 PM Daniel Iwan <iwan.daniel@gmail.com> wrote:
Hi
I'm using Ceph Pacific 16.2.1
I'm creating a topic as a user which belongs to a non-default tenant. I'm using AWS CLI 2 with v3 authentication enabled
aws --profile=ceph-myprofile --endpoint=$HOST_S3_API --region="" sns create-topic --name=fishtopic --attributes='{"push-endpoint": " http://my-ceph-source-svc.default.svc.cluster.local"}' { "TopicArn": "arn:aws:sns:default::fishtopic" }
topic is created in default tenant though. User can list topics but see topics from the default tenant.
aws --profile=ceph-myprofile --endpoint=$HOST_S3_API --region="" sns list-topics { "Topics": [ { "TopicArn": "arn:aws:sns:default::fishtopic" } ] }
Topic is in default tenant
# radosgw-admin topic list --uid none { "topics": [ { "topic": { "user": "", "name": "fishtopic", "dest": { "bucket_name": "", "oid_prefix": "", "push_endpoint": " http://my-ceph-source-svc.default.svc.cluster.local", "push_endpoint_args": "Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value= http://my-ceph-source-svc.default.svc.cluster.local &Version=2010-03-31&push-endpoint= http://my-ceph-source-svc.default.svc.cluster.local", "push_endpoint_topic": "fishtopic", "stored_secret": "false", "persistent": "false" }, "arn": "arn:aws:sns:default::fishtopic", "opaqueData": "" }, "subs": [] } ] }
this looks like a bug, the topic should be created in the right tenant. please submit a tracker for that.
When I create a topic over HTTP with a federated user, the topic is created in the correct (user's) tenant. For some reason the "user" below is "marvel", which is actually the name of the tenant. Possibly the topic is not owned by the user but rather the tenant.
radosgw-admin topic list --tenant marvel --uid none { "topics": [ { "topic": { "user": "marvel", "name": "MyTopic", "dest": { "bucket_name": "", "oid_prefix": "", "push_endpoint": "amqp://127.0.0.1", "push_endpoint_args": "amqp-exchange=rgw-exchange&push-endpoint=amqp://127.0.0.1 &use-ssl=false&verify-ssl=false", "push_endpoint_topic": "MyTopic", "stored_secret": "false", "persistent": "false" }, "arn": "arn:aws:sns:default:marvel:MyTopic", "opaqueData": "" }, "subs": [] } ] }
Also, what permissions are checked when creating a topic? It seems so far I can create a topic without granting any special permissions?
no permissions are needed to create a topic. however, note that without
yes. topics are owned by the tenant. previously, they were owned by the user but since the same topic could be used among different buckets and different users, this was causing issues (was fixed here: https://github.com/ceph/ceph/pull/38136) (documentation also mentioned that in the intro paragraph of the doc: https://docs.ceph.com/en/latest/radosgw/notifications/) for compatibility reasons, the name in the JSON structure still says "user" :-( proper permissions on the bucket, you cannot create a notification that associates this topic with the bucket.
Regards Daniel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
this looks like a bug, the topic should be created in the right tenant. please submit a tracker for that.
Thank you for confirming. Created here https://tracker.ceph.com/issues/51331
yes. topics are owned by the tenant. previously, they were owned by the user but since the same topic could be used among different buckets and different users, this was causing issues (was fixed here: https://github.com/ceph/ceph/pull/38136) (documentation also mentioned that in the intro paragraph of the doc: https://docs.ceph.com/en/latest/radosgw/notifications/)
I think it's this section ``` A user can create different topics. A topic entity is defined by its name and is per tenant. A user can only associate its topics (via notification configuration) with buckets it owns. ```
no permissions are needed to create a topic. however, note that without proper permissions on the bucket, you cannot create a notification that associates this topic with the bucket.
Yes, I thought it would be similar to AWS, possibly not implemented/needed so far: https://docs.aws.amazon.com/sns/latest/dg/sns-using-identity-based-policies.... https://docs.aws.amazon.com/config/latest/developerguide/sns-topic-policy.ht... ``` { "Statement": [{ "Effect": "Allow", "Action": ["sns:CreateTopic", "sns:ListTopics", "sns:SetTopicAttributes", "sns:DeleteTopic"], "Resource": "*" }] } ``` Not having that sns:CreateTopic sns:DeleteTopic leaves room for abuse. User could potentially create many topics, delete all topics from tenant(s) maliciously or by accident (bugs) etc. On a deletion note, if I understand correctly, deletion of the topic without deletion of all notifications first creates the situation where notifications can no longer be deleted due to the topic missing. The only option is to re-create the topic and delete notifications first. Btw I enjoyed your FOSDEM presentation https://fosdem.org/2021/schedule/event/sds_ceph_rgw_serverless/ Any timeframe for native SQS coming to Ceph? Regards Daniel
On Wed, Jun 23, 2021 at 6:39 PM Daniel Iwan <iwan.daniel@gmail.com> wrote:
this looks like a bug, the topic should be created in the right tenant.
please submit a tracker for that.
Thank you for confirming. Created here https://tracker.ceph.com/issues/51331
thanks
yes. topics are owned by the tenant. previously, they were owned by the
user but since the same topic could be used among different buckets and different users, this was causing issues (was fixed here: https://github.com/ceph/ceph/pull/38136) (documentation also mentioned that in the intro paragraph of the doc: https://docs.ceph.com/en/latest/radosgw/notifications/)
I think it's this section ``` A user can create different topics. A topic entity is defined by its name and is per tenant. A user can only associate its topics (via notification configuration) with buckets it owns. ```
no permissions are needed to create a topic. however, note that without proper permissions on the bucket, you cannot create a notification that associates this topic with the bucket.
Yes, I thought it would be similar to AWS, possibly not implemented/needed so far:
https://docs.aws.amazon.com/sns/latest/dg/sns-using-identity-based-policies....
https://docs.aws.amazon.com/config/latest/developerguide/sns-topic-policy.ht...
``` { "Statement": [{ "Effect": "Allow", "Action": ["sns:CreateTopic", "sns:ListTopics", "sns:SetTopicAttributes", "sns:DeleteTopic"], "Resource": "*" }] } ```
Not having that sns:CreateTopic sns:DeleteTopic leaves room for abuse. User could potentially create many topics, delete all topics from tenant(s) maliciously or by accident (bugs) etc.
"tenant" in the RGW is somewhat equivalent to an "account" in AWS. however, "tenant" does not have all the security aspects that an "account" has. adding that would be much wider in scope than the creation/deletion of topics.
On a deletion note, if I understand correctly, deletion of the topic without deletion of all notifications first creates the situation where notifications can no longer be deleted due to the topic missing. The only option is to re-create the topic and delete notifications first.
according to what i tested, this is not the case. deletion of a topic only prevents the creation of new notifications with that topic. it does not effect the deletion of notifications with that topic, not the actual sending of these notifications. note that we also added a cascade delete process to delete all notifications of a bucket when a bucket is deleted. (it should be in pacific: https://github.com/ceph/ceph/pull/38351)
Btw I enjoyed your FOSDEM presentation https://fosdem.org/2021/schedule/event/sds_ceph_rgw_serverless/
thank you! Any timeframe for native SQS coming to Ceph?
no actual timelines... but it should probably land in the main branch later this year :-)
Regards Daniel
Thanks for clarification
according to what i tested, this is not the case. deletion of a topic only prevents the creation of new notifications with that topic. it does not effect the deletion of notifications with that topic, not the actual sending of these notifications.
note that we also added a cascade delete process to delete all notifications of a bucket when a bucket is deleted. (it should be in pacific: https://github.com/ceph/ceph/pull/38351)
You are right, this is cleaned up as expected. I re-ran my tests and it turned out problems show up after deleting the internal topic (created upon creation of notification). When I did that initially I didn't know about internal topics and in my eagerness I deleted all topics to start over. This triggered an issue with deleting notifications. In my case with topics as below # radosgw-admin topic list --tenant marvel --uid none { "topics": [ { "topic": { "user": "marvel", "name": "MyTopic", "dest": { "bucket_name": "", "oid_prefix": "", "push_endpoint": "amqp://127.0.0.1", "push_endpoint_args": "amqp-exchange=rgw-exchange&push-endpoint=amqp://127.0.0.1 &use-ssl=false&verify-ssl=false", "push_endpoint_topic": "MyTopic", "stored_secret": "false", "persistent": "false" }, "arn": "arn:aws:sns:default:marvel:MyTopic", "opaqueData": "" }, "subs": [] }, { "topic": { "user": "marvel", "name": "bucket-by-thanos_notification_MyTopic", "dest": { "bucket_name": "", "oid_prefix": "", "push_endpoint": "amqp://127.0.0.1", "push_endpoint_args": "amqp-exchange=rgw-exchange&push-endpoint=amqp://127.0.0.1 &use-ssl=false&verify-ssl=false", "push_endpoint_topic": "MyTopic", "stored_secret": "false", "persistent": "false" }, "arn": "arn:aws:sns:default:marvel:MyTopic", "opaqueData": "" }, "subs": [] } ] } Once I delete both of those and attempt to delete the notification on the bucket, RGW logs as below. Returned status code is 200 but notification can still be listed. Log debug 2021-06-25T08:36:59.823+0000 7fefed33a700 10 cache get: name=my-rgw.rgw.log++pubsub.marvel.bucket.bucket-by-thanos/103132f8-3d78-4592-ae7b-067ecf2a1b92.9081601.36 : hit (requested=0x6, cached=0x17) debug 2021-06-25T08:36:59.823+0000 7fefed33a700 20 get_system_obj_state: s->obj_tag was set empty debug 2021-06-25T08:36:59.823+0000 7fefed33a700 10 cache get: name=my-rgw.rgw.log++pubsub.marvel.bucket.bucket-by-thanos/103132f8-3d78-4592-ae7b-067ecf2a1b92.9081601.36 : hit (requested=0x1, cached=0x17) debug 2021-06-25T08:36:59.823+0000 7fefed33a700 20 get_system_obj_state: rctx=0x558dd865b1c0 obj=my-rgw.rgw.log:pubsub.marvel state=0x558dd5ff9720 s->prefetch_data=0 debug 2021-06-25T08:36:59.823+0000 7fefed33a700 10 cache get: name=my-rgw.rgw.log++pubsub.marvel : hit (requested=0x16, cached=0x17) debug 2021-06-25T08:36:59.823+0000 7fefed33a700 20 get_system_obj_state: s->obj_tag was set empty debug 2021-06-25T08:36:59.823+0000 7fefed33a700 10 cache get: name=my-rgw.rgw.log++pubsub.marvel : hit (requested=0x11, cached=0x17) debug 2021-06-25T08:36:59.823+0000 7fefed33a700 10 distributing notification oid=my-rgw.rgw.control:notify.0 bl.length()=189 debug 2021-06-25T08:36:59.823+0000 7ff108570700 10 RGWWatcher::handle_notify() notify_id 3573413718119 cookie 94067650647552 notifier 14638216 bl.length()=189 debug 2021-06-25T08:36:59.831+0000 7fefe3b27700 20 get_system_obj_state: rctx=0x558dd865b1c0 obj=my-rgw.rgw.log:pubsub.marvel state=0x558dd5ff9720 s->prefetch_data=0 debug 2021-06-25T08:36:59.831+0000 7fefe3b27700 10 cache get: name=my-rgw.rgw.log++pubsub.marvel : hit (requested=0x1, cached=0x17) debug 2021-06-25T08:36:59.839+0000 7fefe3b27700 1 ERROR: topic not found debug 2021-06-25T08:36:59.839+0000 7fefe3b27700 1 ERROR: failed to read topic info: ret=-2 debug 2021-06-25T08:36:59.839+0000 7fefe3b27700 1 failed to remove notification of topic 'bucket-by-thanos_notification_MyTopic', ret=-2 debug 2021-06-25T08:36:59.839+0000 7fefe3b27700 20 get_system_obj_state: rctx=0x558dd865b1c0 obj=my-rgw.rgw.log:pubsub.marvel state=0x558dd5ff9720 s->prefetch_data=0 debug 2021-06-25T08:36:59.839+0000 7fefe3b27700 10 cache get: name=my-rgw.rgw.log++pubsub.marvel : hit (requested=0x11, cached=0x17) debug 2021-06-25T08:36:59.843+0000 7ff007b6f700 10 cache put: name=my-rgw.rgw.log++pubsub.marvel info.flags=0x17 debug 2021-06-25T08:36:59.843+0000 7ff007b6f700 10 moving my-rgw.rgw.log++pubsub.marvel to cache LRU end debug 2021-06-25T08:36:59.843+0000 7ff007b6f700 10 distributing notification oid=my-rgw.rgw.control:notify.4 bl.length()=189 debug 2021-06-25T08:36:59.843+0000 7ff108d71700 10 RGWWatcher::handle_notify() notify_id 3573413545888 cookie 94067664846848 notifier 14638216 bl.length()=189 debug 2021-06-25T08:36:59.843+0000 7ff108d71700 10 cache put: name=my-rgw.rgw.log++pubsub.marvel info.flags=0x17 debug 2021-06-25T08:36:59.843+0000 7ff108d71700 10 moving my-rgw.rgw.log++pubsub.marvel to cache LRU end debug 2021-06-25T08:36:59.843+0000 7feff6b4d700 2 req 329 0.019999718s s3:pubsub_notification_delete_s3 completing debug 2021-06-25T08:36:59.843+0000 7feff6b4d700 20 get_system_obj_state: rctx=0x7ff122a72750 obj=my-rgw.rgw.log:script.postrequest.marvel state=0x558dd5ff9720 s->prefetch_data=0 debug 2021-06-25T08:36:59.843+0000 7feff6b4d700 10 cache get: name=my-rgw.rgw.log++script.postrequest.marvel : hit (negative entry) debug 2021-06-25T08:36:59.843+0000 7feff6b4d700 2 req 329 0.019999718s s3:pubsub_notification_delete_s3 op status=0 debug 2021-06-25T08:36:59.847+0000 7feff6b4d700 2 req 329 0.023999661s s3:pubsub_notification_delete_s3 http status=200 debug 2021-06-25T08:36:59.847+0000 7feff6b4d700 1 ====== req done req=0x7ff122a73620 op status=0 http_status=200 latency=0.023999661s ====== debug 2021-06-25T08:36:59.847+0000 7feff6b4d700 1 beast: 0x7ff122a73620: 11.1.150.14 - marvel$oidc$f1be270a-d6a0-4f97-b444-c1f23816b25a [25/Jun/2021:08:36:59.823 +0000] "DELETE /bucket-by-thanos?notification=bucket-by-thanos_notification HTTP/1.1" 200 0 - "PostmanRuntime/7.28.0" - Thanks Daniel
participants (2)
-
Daniel Iwan
-
Yuval Lifshitz