bucket notification retries
Dear Community, I would like to collect your feedback on this issue. This is a followup from a discussion that started in the RGW refactoring meeting on 31-May-23 (thanks @Krunal Chheda <kchheda3@bloomberg.net> for bringing up this topic!). Currently persistent notifications are retried indefinitely. The only limiting mechanism that exists is that all notifications to a specific topic are stored in one RADOS object (of size 128MB). Assuming notifications are ~1KB at most, this would give us at least 128K notifications that can wait in the queue. When the queue fills up (e.g. kafka broker is down for 20 minutes, we are sending ~100 notifications per second) we start sending "slow down" replies to the client, and in this case the S3 operation will not be performed. This means that, for example, an outage of the kafka system would eventually cause an outage of our service. Note that this may also be a result of a misconfiguration of the kafka broker, or decommissioning of a broker. To avoid that, we propose several options: * use a fifo instead of a queue. This would allow us to hold more than 128K messages - survive longer broker outages, and at a higher message rate. there should still probably be a limit set on the size of the fifo * define maximum number of retries allowed for a notification * define maximum time the notification may stay in the queue before it is removed We should probably start with these definitions done as topic attributes, reflecting our delivery guarantees for this specific destination. Will try to capture the results of the discussion in this tracker: https://tracker.ceph.com/issues/61532 Thanks, Yuval
Hi Yuval, Thanks for having a look at bucket notifications and collecting feedback. I also see potential for improvement in the area of bucket notifications. We have observed issues in a setup with Rabbit MQ as a broker where the RADOS queue seems to fill up and cients receive "slow down" replies. Unfortunately this state did not recover. The only solution to overcome the situation was to remove and recreate the topic and bucket notification configuration. This happened multiple times on differenct ceph clusters with latest quincy. It would be great to improve the ability to monitor bucket notifications (e.g. via prometheus/grafana) to see the RADOS queues and their usage/queue depth as well as the health of the process that consumes the queue and passes the notifications to the broker. For our use case notifications are very important as they trigger downstream processing of the uploaded files. If the notification does not happen, the files are not processed and the result is the same as if the upload did not happen at all. Best regards, Stefan
Hi Stefan, Thanks for the inputs. Replied inline On Fri, Jun 9, 2023 at 6:53 PM Stefan Reuter <stefan.reuter@reucon.com> wrote:
Hi Yuval,
Thanks for having a look at bucket notifications and collecting feedback. I also see potential for improvement in the area of bucket notifications.
We have observed issues in a setup with Rabbit MQ as a broker where the RADOS queue seems to fill up and cients receive "slow down" replies. Unfortunately this state did not recover. The only solution to overcome the situation was to remove and recreate the topic and bucket notification configuration. This happened multiple times on differenct ceph clusters with latest quincy.
will check that. We had a similar issue with Kafka broker that was recently fixed.
It would be great to improve the ability to monitor bucket notifications (e.g. via prometheus/grafana) to see the RADOS queues and their usage/queue depth as well as the health of the process that consumes the queue and passes the notifications to the broker.
agree. we are working on that. see: https://tracker.ceph.com/issues/52927
For our use case notifications are very important as they trigger downstream processing of the uploaded files. If the notification does not happen, the files are not processed and the result is the same as if the upload did not happen at all.
Best regards,
Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Stefan, I was not able to reproduce the issue of not reconnecting after slow-down. My steps are documented here: https://gist.github.com/yuvalif/e58e264bafe847bc5196f95be0e704a2 Can you please share some of the radosgw logs after the broker is up again and the reconnect fails? Regardless, there are several race conditions that happened with kafka and persistent notifications and also exists for amqp. Will be fixing that as part of: https://tracker.ceph.com/issues/61639 Yuval On Sun, Jun 11, 2023 at 11:48 AM Yuval Lifshitz <ylifshit@redhat.com> wrote:
Hi Stefan, Thanks for the inputs. Replied inline
On Fri, Jun 9, 2023 at 6:53 PM Stefan Reuter <stefan.reuter@reucon.com> wrote:
Hi Yuval,
Thanks for having a look at bucket notifications and collecting feedback. I also see potential for improvement in the area of bucket notifications.
We have observed issues in a setup with Rabbit MQ as a broker where the RADOS queue seems to fill up and cients receive "slow down" replies. Unfortunately this state did not recover. The only solution to overcome the situation was to remove and recreate the topic and bucket notification configuration. This happened multiple times on differenct ceph clusters with latest quincy.
will check that. We had a similar issue with Kafka broker that was recently fixed.
It would be great to improve the ability to monitor bucket notifications (e.g. via prometheus/grafana) to see the RADOS queues and their usage/queue depth as well as the health of the process that consumes the queue and passes the notifications to the broker.
agree. we are working on that. see: https://tracker.ceph.com/issues/52927
For our use case notifications are very important as they trigger downstream processing of the uploaded files. If the notification does not happen, the files are not processed and the result is the same as if the upload did not happen at all.
Best regards,
Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Stefan Reuter
-
Yuval Lifshitz