Reef: RGW Multisite object fetch limits
Hello Community, We've two zones with Reef (v18.2.1) and trying to sync over 2 billion RGW objects to the secondary zone. We've added a fresh secondary zone and each zones have 2 RGW dedicated daemons (behind LB) each and only for multisite; whereas others don't run sync threads. Strange thing is that during the sync, the master zone LB records only 1k to 1.5k HTTP requests per second from secondary zone RGWs and not more than that. This behaviour leaves us with less link utilization. I don't see any noticeable issues with the link, cluster or RGW daemons on both ends. We've already increased rgw_data_sync_spawn_window, rgw_bucket_sync_spawn_window, rgw_meta_sync_spawn_window values to more than the defaults and don't seem to improve. Has someone noticed this behaviour of secondary zone RGW daemons fetching only around 1k to 1.5k objects per second? Regards, Jayanth
Hello Community, In addition, we've 3+ Gbps links and the average object size is 200 kilobytes. So the utilization is about 300 Mbps to ~ 1.8 Gbps and not more than that. We seem to saturate the link when the secondary zone fetches bigger objects sometimes but the objects per second always seem to be 1k to 1.5k per second. Regards, Jayanth On Thu, May 16, 2024 at 11:05 AM Jayanth Reddy <jayanthreddy5666@gmail.com> wrote:
Hello Community, We've two zones with Reef (v18.2.1) and trying to sync over 2 billion RGW objects to the secondary zone. We've added a fresh secondary zone and each zones have 2 RGW dedicated daemons (behind LB) each and only for multisite; whereas others don't run sync threads. Strange thing is that during the sync, the master zone LB records only 1k to 1.5k HTTP requests per second from secondary zone RGWs and not more than that. This behaviour leaves us with less link utilization. I don't see any noticeable issues with the link, cluster or RGW daemons on both ends. We've already increased rgw_data_sync_spawn_window, rgw_bucket_sync_spawn_window, rgw_meta_sync_spawn_window values to more than the defaults and don't seem to improve.
Has someone noticed this behaviour of secondary zone RGW daemons fetching only around 1k to 1.5k objects per second?
Regards, Jayanth
Den tors 16 maj 2024 kl 07:47 skrev Jayanth Reddy <jayanthreddy5666@gmail.com>:
Hello Community, In addition, we've 3+ Gbps links and the average object size is 200 kilobytes. So the utilization is about 300 Mbps to ~ 1.8 Gbps and not more than that. We seem to saturate the link when the secondary zone fetches bigger objects sometimes but the objects per second always seem to be 1k to 1.5k per second.
Is it possible that the small object sizes makes it impossible for the replication to get any decent speed? If it makes a new tcp connection for every S3 object, then round-trip-times and the small sizes would make it impossible to get up to decent speed over the network before the object is finished, and then it restarts again with a new object with a new slow start and so on. -- May the most significant bit of your life be positive.
Correct. Nagle's algorithm has been disabled on RGWs and LBs but not seeing any significant difference, and also looks like RGWs don't support HTTP/2. In addition to this, we're also being affected by [1]. [1] https://tracker.ceph.com/issues/64999 Regards, Jayanth On Thu, May 16, 2024 at 2:40 PM Janne Johansson <icepic.dz@gmail.com> wrote:
Den tors 16 maj 2024 kl 07:47 skrev Jayanth Reddy < jayanthreddy5666@gmail.com>:
Hello Community, In addition, we've 3+ Gbps links and the average object size is 200 kilobytes. So the utilization is about 300 Mbps to ~ 1.8 Gbps and not
more
than that. We seem to saturate the link when the secondary zone fetches bigger objects sometimes but the objects per second always seem to be 1k to 1.5k per second.
Is it possible that the small object sizes makes it impossible for the replication to get any decent speed?
If it makes a new tcp connection for every S3 object, then round-trip-times and the small sizes would make it impossible to get up to decent speed over the network before the object is finished, and then it restarts again with a new object with a new slow start and so on.
-- May the most significant bit of your life be positive.
participants (2)
-
Janne Johansson
-
Jayanth Reddy