Re: Sending notification after multiple objects are created in a ceph bucket.
Facing a similar situation, any support would be helpful. -Lokendra On Tue, Jan 9, 2024 at 10:47 PM Kushagr Gupta <kushagrguptasps.mun@gmail.com> wrote:
Hi Team,
Features used: Rados gateway, ceph S3 buckets
We are trying to create a data pipeline using the S3 buckets capability and rado gateway in ceph. Our endpoint is a kafka topic.
Currently, as soon as an object is created, a notification is sent to kafka topic. Our goal is to create multiple object first and then send the notification. Also if there is a way such that we can assign objects to a group and collectively send notification on an event on that group.
Could anyone please help me?
Thanks and Regards, Kushagra Gupta
-- ~ Lokendra skype: lokendrarathour
Lokendra and Kushagra, We don't have such an enhancement on the roadmap. Would think of 2 options: (1) implement the special logic using lua scripting. We have an example on how to send notifications to a NATS broker from lua [1]. you can easily adjust that to kafka. the 2 main drawbacks with this solution are: * performance and latency, everything done in the lua script is synchronous with the S3 operation * lua scripts have the "global" RGW table for aggregations. however, this is local to an RGW and not shared across RGWs. in addition, the information there is not persistent, and will be lost if the RGW crash/restart (2) implement that externally to the RGW. e.g. use bucket notifications to trigger a serverless function (probably backed with some db) that does the aggregation and send the notifications according to the logic that you want. see [2] also, there is the possibility of developing that feature and contributing it to ceph. note that it would be a considerably more effort than the options above. Yuval [1] https://github.com/ceph/ceph/blob/main/examples/rgw/lua/nats_adapter.md [2] https://www.youtube.com/watch?v=1E5Q-ErF2sI On Thu, Jan 11, 2024 at 11:08 AM Lokendra Rathour <lokendrarathour@gmail.com> wrote:
Facing a similar situation, any support would be helpful.
-Lokendra
On Tue, Jan 9, 2024 at 10:47 PM Kushagr Gupta < kushagrguptasps.mun@gmail.com> wrote:
Hi Team,
Features used: Rados gateway, ceph S3 buckets
We are trying to create a data pipeline using the S3 buckets capability and rado gateway in ceph. Our endpoint is a kafka topic.
Currently, as soon as an object is created, a notification is sent to kafka topic. Our goal is to create multiple object first and then send the notification. Also if there is a way such that we can assign objects to a group and collectively send notification on an event on that group.
Could anyone please help me?
Thanks and Regards, Kushagra Gupta
-- ~ Lokendra skype: lokendrarathour _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Lokendra Rathour
-
Yuval Lifshitz