RGW listing slower on nominally faster setup
Hi everyone, We are currently transitioning from a temporary machine to our production hardware. Since we're starting with under 200 TB raw storage, we are currently on only 1–2 physical machines per cluster, eventually in 3 zones. The temporary machine is undersized for even that with an older single 6-core CPU and spinning disks only. As of now that "cluster-of-one" is running on Nautilus and has 3 buckets with 98K, 1.1M and 1.4M objects, respectively for a total of 9.1 TB. As we're expecting these to grow to around 5M objects each and will be in a multisite configuration, I went with 50 shards per bucket. Listing "directories" via S3 is somewhat slow (sometimes to the point of read timeouts) but mostly bearable. After the new production setup (dual 8-core/16-thread Xeon Silvers, 2 x SATA SSDs for RGW index pool, on Octopus, with enough free memory to easily fit all bucket indexes multiple times) synced successfully, listings via S3 always time out on the RGW on that machine/zone. As soon as I trigger a single listing via S3 (even on the 98K object bucket), reads go up to a sustained 300–500MB/s and 20–50K IOPS on the bucket index pool for several hours. The RGW debug log is flooded with lines like this: {"log":"debug 2020-06-08T19:31:08.315+0000 7f83d704c700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1\n","stream":"stdout","time":"2020-06-08T19:31:08.317198682Z"} I get that sharded RGW indexes (and listing objects in S3 buckets in general) are not very efficient, but after getting somewhat decent results on slower hardware and an older Ceph version, I wasn't expecting the nominally much better setup to be orders of magnitude slower. Any help or pointers would be greatly appreciated. Thank you, Stefan
Stefan; I can't find it, but I seem to remember a discussion in this mailing list that sharded RGW performance is significantly better if the shard count is a power of 2, so you might try increasing shards to 64. Also, you might looks at OSD logs while a listing is trying to run, to see if this illuminates anything for you. You said: "2 x SATA SSDs for RGW index pool," but do you have the zone's index pool running on a rule which only targets SSDs, or only targets those SSDs? Are you running your RGW multi-site? Are you running replication for RGW in multi-site? Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International, Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Stefan Wild [mailto:swild@tiltworks.com] Sent: Wednesday, June 10, 2020 6:05 PM To: ceph-users@ceph.io Subject: [ceph-users] RGW listing slower on nominally faster setup Hi everyone, We are currently transitioning from a temporary machine to our production hardware. Since we're starting with under 200 TB raw storage, we are currently on only 1–2 physical machines per cluster, eventually in 3 zones. The temporary machine is undersized for even that with an older single 6-core CPU and spinning disks only. As of now that "cluster-of-one" is running on Nautilus and has 3 buckets with 98K, 1.1M and 1.4M objects, respectively for a total of 9.1 TB. As we're expecting these to grow to around 5M objects each and will be in a multisite configuration, I went with 50 shards per bucket. Listing "directories" via S3 is somewhat slow (sometimes to the point of read timeouts) but mostly bearable. After the new production setup (dual 8-core/16-thread Xeon Silvers, 2 x SATA SSDs for RGW index pool, on Octopus, with enough free memory to easily fit all bucket indexes multiple times) synced successfully, listings via S3 always time out on the RGW on that machine/zone. As soon as I trigger a single listing via S3 (even on the 98K object bucket), reads go up to a sustained 300–500MB/s and 20–50K IOPS on the bucket index pool for several hours. The RGW debug log is flooded with lines like this: {"log":"debug 2020-06-08T19:31:08.315+0000 7f83d704c700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1\n","stream":"stdout","time":"2020-06-08T19:31:08.317198682Z"} I get that sharded RGW indexes (and listing objects in S3 buckets in general) are not very efficient, but after getting somewhat decent results on slower hardware and an older Ceph version, I wasn't expecting the nominally much better setup to be orders of magnitude slower. Any help or pointers would be greatly appreciated. Thank you, Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the tip on power of 2 shard numbers. I want to say I've read that, too. We have a small window where I can re-shard the index before we sync the other 2 zones. Will check on OSD logs, as well. As to the SSDs – yes, I've confirmed that the bucket index is placed only on those two SSDs via a 2 x replicated rule.
Not seeing anything in OSD logs after triggering a listing, just heartbeat entries.
Hi, I’m experiencing the same symptoms as OP. We’re running Ceph Octopus 15.2.1 with RGW, and have seen on multiple occasions the bucket index pool go up to 500MB/s read throughput / 100K read IOPS. Our logs during this time are flooded with these entries: 2020-06-09T07:11:18.070+0200 7f2676efd700 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 When I set the debug_rgw logs to "20/1", the issue disappears immediately, and the throughput for the index pool goes back down to normal levels. I’ve not actually tried reproducing the issue myself as I assumed the problem was with the S3 client, but maybe this is a bug on the RGW side… Our bucket index pool is running on the same HDDs as the data pool, no separate SSDs like OP. Regards, James
On 12 Jun 2020, at 02:22, swild@tiltworks.com wrote:
Not seeing anything in OSD logs after triggering a listing, just heartbeat entries. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 6/12/20, 5:40 AM, "James, GleSYS" <james.mcewan@glesys.se> wrote:
When I set the debug_rgw logs to "20/1", the issue disappears immediately, and the throughput for the index pool goes back down to normal levels.
I can – somewhat happily – confirm that setting debug_rgw to "20/1" makes the issue disappear instantly. Even if the RGW is in the middle of a "stuck" listing, the debug level change causes the load to drop and results appear on the client almost instantly. After setting debug_rgw back to "5/1" the listings get stuck again, which in our case is not just occasionally, but always and for every bucket. Not exactly a solution, since we're already having some trouble keeping the docker container logs to a manageable size, but might be good enough as a workaround. Not sure how we can find out steps to reproduce the issue in the first place. Happy to do some testing if anyone has suggestions. Also, I'm inclined to report this as a bug at this point unless there's opposing advice. Thanks, Stefan
hello is there a way to push this config directly into ceph without using the ceph.conf file ? thanks for your tips oau Le vendredi 12 juin 2020 à 15:24 +0000, Stefan Wild a écrit :
On 6/12/20, 5:40 AM, "James, GleSYS" <james.mcewan@glesys.se> wrote:
When I set the debug_rgw logs to "20/1", the issue disappears immediately, and the throughput for the index pool goes back down to normal levels.
I can – somewhat happily – confirm that setting debug_rgw to "20/1" makes the issue disappear instantly. Even if the RGW is in the middle of a "stuck" listing, the debug level change causes the load to drop and results appear on the client almost instantly. After setting debug_rgw back to "5/1" the listings get stuck again, which in our case is not just occasionally, but always and for every bucket. Not exactly a solution, since we're already having some trouble keeping the docker container logs to a manageable size, but might be good enough as a workaround.
Not sure how we can find out steps to reproduce the issue in the first place. Happy to do some testing if anyone has suggestions. Also, I'm inclined to report this as a bug at this point unless there's opposing advice.
Thanks, Stefan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We have a cluster, running Octopus 15.2.2, with the same exact issue described originally. Confirmed, setting debug_rgw logs to "20/1" fixed the issue for us as well. What information would be needed to begin a preliminary bug report? As with others, I haven't found a way to easily replicate this issue. I can confirm this issue was also not occasional for us, but also affected every bucket, always, like Stefan saw.
I've already filled a bug for it as we ran into same issue: https://tracker.ceph.com/issues/45955 might want to add that extra info about debug_rgw. Dnia 2020-06-24, o godz. 20:31:35 jgoetz@teraswitch.com napisał(a):
We have a cluster, running Octopus 15.2.2, with the same exact issue described originally.
Confirmed, setting debug_rgw logs to "20/1" fixed the issue for us as well.
What information would be needed to begin a preliminary bug report? As with others, I haven't found a way to easily replicate this issue. I can confirm this issue was also not occasional for us, but also affected every bucket, always, like Stefan saw. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Mariusz Gronczewski, Administrator Efigence S. A. ul. Wołoska 9a, 02-583 Warszawa T: [+48] 22 380 13 13 NOC: [+48] 22 380 10 20 E: admin@efigence.com
participants (7)
-
DHilsbos＠performair.com
-
James, GleSYS
-
jgoetz＠teraswitch.com
-
Mariusz Gronczewski
-
Olivier AUDRY
-
Stefan Wild
-
swild＠tiltworks.com