Adventures with large RGW buckets
Hi, we are seeing a trend towards rather large RGW S3 buckets lately. we've worked on several clusters with 100 - 500 million objects in a single bucket, and we've been asked about the possibilities of buckets with several billion objects more than once. From our experience: buckets with tens of million objects work just fine with no big problems usually. Buckets with hundreds of million objects require some attention. Buckets with billions of objects? "How about indexless buckets?" - "No, we need to list them". A few stories and some questions: 1. The recommended number of objects per shard is 100k. Why? How was this default configuration derived? It doesn't really match my experiences. We know a few clusters running with larger shards because resharding isn't possible for various reasons at the moment. They sometimes work better than buckets with lots of shards. So we've been considering to at least double that 100k target shard size for large buckets, that would make the following point far less annoying. 2. Many shards + ordered object listing = lots of IO Unfortunately telling people to not use ordered listings when they don't really need them doesn't really work as their software usually just doesn't support that :( A listing request for X objects will retrieve up to X objects from each shard for ordering them. That will lead to quite a lot of traffic between the OSDs and the radosgw instances, even for relatively innocent simple queries as X defaults to 1000 usually. Simple example: just getting the first page of a bucket listing with 4096 shards fetches around 1 GB of data from the OSD to return ~300kb or so to the S3 client. I've got two clusters here that are only used for some relatively low-bandwidth backup use case here. However, there are a few buckets with hundreds of millions of objects that are sometimes being listed by the backup system. The result is that this cluster has an average read IO of 1-2 GB/s, all going to the index pool. Not a big deal since that's coming from SSDs and goes over 80 Gbit/s LACP bonds. But it does pose the question about scalability as the user- visible load created by the S3 clients is quite low. 3. Deleting large buckets Someone accidentaly put 450 million small objects into a bucket and only noticed when the cluster ran full. The bucket isn't needed, so just delete it and case closed? Deleting is unfortunately far slower than adding objects, also radosgw-admin leaks memory during deletion: https://tracker.ceph.com/issues/40700 Increasing --max-concurrent-ios helps with deletion speed (option does effect deletion concurrency, documentation says it's only for other specific commands). Since the deletion is going faster than new data is being added to that cluster the "solution" was to run the deletion command in a memory-limited cgroup and restart it automatically after it gets killed due to leaking. How could the bucket deletion of the future look like? Would it be possible to put all objects in buckets into RADOS namespaces and implement some kind of efficient namespace deletion on the OSD level similar to how pool deletions are handled at a lower level? 4. Common prefixes could filtered in the rgw class on the OSD instead of in radosgw Consider a bucket with 100 folders with 1000 objects in each and only one shard /p1/1, /p1/2, ..., /p1/1000, /p2/1, /p2/2, ..., /p2/1000, ... /p100/1000 Now a user wants to list / with aggregating common prefixes on the delimiter / and wants up to 1000 results. So there'll be 100 results returned to the client: the common prefixes p1 to p100. How much data will be transfered between the OSDs and radosgw for this request? How many omap entries does the OSD scan? radosgw will ask the (single) index object to list the first 1000 objects. It'll return 1000 objects in a quite unhelpful way: /p1/1, /p1/2, ...., /p1/1000 radosgw will discard 999 of these and detect one common prefix and continue the iteration at /p1/\xFF to skip the remaining entries in /p1/ if there are any. The OSD will then return everything in /p2/ in that next request and so on. So it'll internally list every single object in that bucket. That will be a problem for large buckets and having lots of shards doesn't help either. This shouldn't be too hard to fix: add an option "aggregate prefixes" to the RGW class method and duplicate the fast-forward logic from radosgw in cls_rgw. It doesn't even need to change the response type or anything, it just needs to limit entries in common prefixes to one result. Is this a good idea or am I missing something? IO would be reduced by a factor of 100 for that particular pathological case. I've unfortunately seen a real-world setup that I think hits a case like that. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90
Hi, On 31/07/2019 19:02, Paul Emmerich wrote: Some interesting points here, thanks for raising them :)
From our experience: buckets with tens of million objects work just fine with no big problems usually. Buckets with hundreds of million objects require some attention. Buckets with billions of objects? "How about indexless buckets?" - "No, we need to list them".
We've had some problems with large buckets (from around the 70Mobject mark). One you don't mention is that multipart uploads break during resharding - so if our users are filling up a bucket with many writers uploading multipart objects, some of these will fail (rather than blocking) when the bucket is resharded.
1. The recommended number of objects per shard is 100k. Why? How was this default configuration derived?
I don't know what a good number is, but by the time you get into O(10M) objects, some sharding does seem to help - we've found a particular OSD getting really hammered by heavy updates on large buckets (in Jewel, before we had online resharding).
3. Deleting large buckets
Someone accidentaly put 450 million small objects into a bucket and only noticed when the cluster ran full. The bucket isn't needed, so just delete it and case closed?
Deleting is unfortunately far slower than adding objects, also radosgw-admin leaks memory during deletion:
We've also seen bucket deletion via radosgw-admin failing because of oddities in the bucket itself (e.g. missing shadow objects, omap objects that still exist when the related object is gone); sorting that was a bit fiddly (with some help from Canonical, who I think are working on patches).
Increasing --max-concurrent-ios helps with deletion speed (option does effect deletion concurrency, documentation says it's only for other specific commands).
Yes, we found increasing max-concurrent-ios helped. Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
On 2019-08-01T15:20:19, Matthew Vernon <mv3@sanger.ac.uk> wrote:
One you don't mention is that multipart uploads break during resharding - so if our users are filling up a bucket with many writers uploading multipart objects, some of these will fail (rather than blocking) when the bucket is resharded.
Is that on the tracker? I couldn't find it. If you can reproduce, would you add that please? (I found https://tracker.ceph.com/issues/22368 and https://tracker.ceph.com/issues/38486, which may be related.) Thanks, Lars -- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
Hi, On 02/08/2019 13:23, Lars Marowsky-Bree wrote:
On 2019-08-01T15:20:19, Matthew Vernon <mv3@sanger.ac.uk> wrote:
One you don't mention is that multipart uploads break during resharding - so if our users are filling up a bucket with many writers uploading multipart objects, some of these will fail (rather than blocking) when the bucket is resharded.
Is that on the tracker? I couldn't find it. If you can reproduce, would you add that please?
Not as yet - our support vendor have reproduced the issue, so I'll ask them to open a ticket on the tracker (I mean, I could, but their reproducer is probably neater than mine :) ). Regards, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
A few interleaved responses below.... On 8/1/19 10:20 AM, Matthew Vernon wrote:
Hi,
On 31/07/2019 19:02, Paul Emmerich wrote:
Some interesting points here, thanks for raising them :)
We've had some problems with large buckets (from around the 70Mobject mark).
One you don't mention is that multipart uploads break during resharding - so if our users are filling up a bucket with many writers uploading multipart objects, some of these will fail (rather than blocking) when the bucket is resharded.
Is there a tracker for that already? If not, would you mind adding one?
We've also seen bucket deletion via radosgw-admin failing because of oddities in the bucket itself (e.g. missing shadow objects, omap objects that still exist when the related object is gone); sorting that was a bit fiddly (with some help from Canonical, who I think are working on patches).
There was a recently merged PR that addressed bucket deletion with missing shadow objects: https://tracker.ceph.com/issues/40590 Thank you for reporting your experience w/ rgw, Eric -- J. Eric Ivancich he/him/his Red Hat Storage Ann Arbor, Michigan, USA
Hi Paul, I’ll interleave responses below.
On Jul 31, 2019, at 2:02 PM, Paul Emmerich <paul.emmerich@croit.io> wrote:
we are seeing a trend towards rather large RGW S3 buckets lately. we've worked on several clusters with 100 - 500 million objects in a single bucket, and we've been asked about the possibilities of buckets with several billion objects more than once.
From our experience: buckets with tens of million objects work just fine with no big problems usually. Buckets with hundreds of million objects require some attention. Buckets with billions of objects? "How about indexless buckets?" - "No, we need to list them".
A few stories and some questions:
1. The recommended number of objects per shard is 100k. Why? How was this default configuration derived?
It doesn't really match my experiences. We know a few clusters running with larger shards because resharding isn't possible for various reasons at the moment. They sometimes work better than buckets with lots of shards.
So we've been considering to at least double that 100k target shard size for large buckets, that would make the following point far less annoying.
I believe the 100,000 objects per shard was done with a little bit of experience and some back-of-the-envelope calculations. Please keep us updated as to what you find for 200,000 objects per shard.
2. Many shards + ordered object listing = lots of IO
Unfortunately telling people to not use ordered listings when they don't really need them doesn't really work as their software usually just doesn't support that :(
We are exploring sharding schemes that maintain ordering, and that would really help here.
A listing request for X objects will retrieve up to X objects from each shard for ordering them. That will lead to quite a lot of traffic between the OSDs and the radosgw instances, even for relatively innocent simple queries as X defaults to 1000 usually.
What you say is correct. And it gets worse, because we have to go through all the returned lists and select the, say, 1000 earliest to return. And then we throw the rest away.
Simple example: just getting the first page of a bucket listing with 4096 shards fetches around 1 GB of data from the OSD to return ~300kb or so to the S3 client.
Correct.
I've got two clusters here that are only used for some relatively low-bandwidth backup use case here. However, there are a few buckets with hundreds of millions of objects that are sometimes being listed by the backup system.
The result is that this cluster has an average read IO of 1-2 GB/s, all going to the index pool. Not a big deal since that's coming from SSDs and goes over 80 Gbit/s LACP bonds. But it does pose the question about scalability as the user- visible load created by the S3 clients is quite low.
3. Deleting large buckets
Someone accidentaly put 450 million small objects into a bucket and only noticed when the cluster ran full. The bucket isn't needed, so just delete it and case closed?
Deleting is unfortunately far slower than adding objects, also radosgw-admin leaks memory during deletion: https://tracker.ceph.com/issues/40700
Increasing --max-concurrent-ios helps with deletion speed (option does effect deletion concurrency, documentation says it's only for other specific commands).
Since the deletion is going faster than new data is being added to that cluster the "solution" was to run the deletion command in a memory-limited cgroup and restart it automatically after it gets killed due to leaking.
That tracker is being investigated.
How could the bucket deletion of the future look like? Would it be possible to put all objects in buckets into RADOS namespaces and implement some kind of efficient namespace deletion on the OSD level similar to how pool deletions are handled at a lower level?
I’ll raise that with other RGW developers. I’m unfamiliar with how RADOS namespaces are handled.
4. Common prefixes could filtered in the rgw class on the OSD instead of in radosgw
Consider a bucket with 100 folders with 1000 objects in each and only one shard
/p1/1, /p1/2, ..., /p1/1000, /p2/1, /p2/2, ..., /p2/1000, ... /p100/1000
Now a user wants to list / with aggregating common prefixes on the delimiter / and wants up to 1000 results. So there'll be 100 results returned to the client: the common prefixes p1 to p100.
How much data will be transfered between the OSDs and radosgw for this request? How many omap entries does the OSD scan?
radosgw will ask the (single) index object to list the first 1000 objects. It'll return 1000 objects in a quite unhelpful way: /p1/1, /p1/2, ...., /p1/1000
radosgw will discard 999 of these and detect one common prefix and continue the iteration at /p1/\xFF to skip the remaining entries in /p1/ if there are any. The OSD will then return everything in /p2/ in that next request and so on.
So it'll internally list every single object in that bucket. That will be a problem for large buckets and having lots of shards doesn't help either.
This shouldn't be too hard to fix: add an option "aggregate prefixes" to the RGW class method and duplicate the fast-forward logic from radosgw in cls_rgw. It doesn't even need to change the response type or anything, it just needs to limit entries in common prefixes to one result. Is this a good idea or am I missing something?
On the face it looks good. I’ll raise this with other RGW developers. I do know that there was a related bug that was recently addressed with this pr: https://github.com/ceph/ceph/pull/28192 <https://github.com/ceph/ceph/pull/28192> But your suggestion seems to go farther.
IO would be reduced by a factor of 100 for that particular pathological case. I've unfortunately seen a real-world setup that I think hits a case like that.
Thank you for sharing your experiences and your ideas. Eric -- J. Eric Ivancich he/him/his Red Hat Storage Ann Arbor, Michigan, USA
On Thu, Aug 1, 2019 at 12:06 PM Eric Ivancich <ivancich@redhat.com> wrote:
Hi Paul,
I’ll interleave responses below.
On Jul 31, 2019, at 2:02 PM, Paul Emmerich <paul.emmerich@croit.io> wrote:
How could the bucket deletion of the future look like? Would it be possible to put all objects in buckets into RADOS namespaces and implement some kind of efficient namespace deletion on the OSD level similar to how pool deletions are handled at a lower level?
I’ll raise that with other RGW developers. I’m unfamiliar with how RADOS namespaces are handled.
I expect RGW could do this, but unfortunately deleting namespaces at the RADOS level is not practical. People keep asking and maybe in some future world it will be cheaper, but a namespace is effectively just part of the object name (and I don't think it's even the first thing they sort by for the key entries in metadata tracking!), so deleting a namespace would be equivalent to deleting a snapshot[1] but with the extra cost that namespaces can be created arbitrarily on every write operation (so our solutions for handling snapshots without it being ludicrously expensive wouldn't apply). Deleting a namespace from the OSD-side using map updates would require the OSD to iterate through just about all the objects they have and examine them for deletion. Is it cheaper than doing over the network? Sure. Is it cheap enough we're willing to let a single user request generate that kind of cluster IO on an unconstrained interface? Absolutely not. -Greg [1]: Deleting snapshots is only feasible because every OSD maintains a sorted secondary index from snapid->set<objects>. This is only possible because snapids are issued by the monitors and clients cooperate in making sure they can't get reused after being deleted. Namespaces are generated by clients and there are no constraints on their use, reuse, or relationship to each other. We could maybe work around these problems, but it'd be building a fundamentally different interface than what namespaces currently are.
HI Greg / Eric, What about allow delete bucket object with a lifecycle policy? You can actually put 1 day of object life, that task is done at cluster level. And them delete objects young than 1 day, and remove bucket. That sometimes speed deletes as task is done by rgw's. It should be like a background delete option, due deleting bucket of millions of objects take weeks. Regards -----Mensaje original----- De: ceph-users <ceph-users-bounces@lists.ceph.com> En nombre de Gregory Farnum Enviado el: jueves, 1 de agosto de 2019 22:48 Para: Eric Ivancich <ivancich@redhat.com> CC: Ceph Users <ceph-users@lists.ceph.com>; dev@ceph.io Asunto: Re: [ceph-users] Adventures with large RGW buckets On Thu, Aug 1, 2019 at 12:06 PM Eric Ivancich <ivancich@redhat.com> wrote:
Hi Paul,
I’ll interleave responses below.
On Jul 31, 2019, at 2:02 PM, Paul Emmerich <paul.emmerich@croit.io> wrote:
How could the bucket deletion of the future look like? Would it be possible to put all objects in buckets into RADOS namespaces and implement some kind of efficient namespace deletion on the OSD level similar to how pool deletions are handled at a lower level?
I’ll raise that with other RGW developers. I’m unfamiliar with how RADOS namespaces are handled.
I expect RGW could do this, but unfortunately deleting namespaces at the RADOS level is not practical. People keep asking and maybe in some future world it will be cheaper, but a namespace is effectively just part of the object name (and I don't think it's even the first thing they sort by for the key entries in metadata tracking!), so deleting a namespace would be equivalent to deleting a snapshot[1] but with the extra cost that namespaces can be created arbitrarily on every write operation (so our solutions for handling snapshots without it being ludicrously expensive wouldn't apply). Deleting a namespace from the OSD-side using map updates would require the OSD to iterate through just about all the objects they have and examine them for deletion. Is it cheaper than doing over the network? Sure. Is it cheap enough we're willing to let a single user request generate that kind of cluster IO on an unconstrained interface? Absolutely not. -Greg [1]: Deleting snapshots is only feasible because every OSD maintains a sorted secondary index from snapid->set<objects>. This is only possible because snapids are issued by the monitors and clients cooperate in making sure they can't get reused after being deleted. Namespaces are generated by clients and there are no constraints on their use, reuse, or relationship to each other. We could maybe work around these problems, but it'd be building a fundamentally different interface than what namespaces currently are. _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
On Thu, Aug 1, 2019 at 10:48 PM Gregory Farnum <gfarnum@redhat.com> wrote:
On Thu, Aug 1, 2019 at 12:06 PM Eric Ivancich <ivancich@redhat.com> wrote: I expect RGW could do this, but unfortunately deleting namespaces at the RADOS level is not practical. People keep asking and maybe in some future world it will be cheaper, but a namespace is effectively just part of the object name (and I don't think it's even the first thing they sort by for the key entries in metadata tracking!), so deleting a namespace would be equivalent to deleting a snapshot[1] but with the extra cost that namespaces can be created arbitrarily on every write operation (so our solutions for handling snapshots without it being ludicrously expensive wouldn't apply). Deleting a namespace from the OSD-side using map updates would require the OSD to iterate through just about all the objects they have and examine them for deletion.
yes, i was thinking of something similar to snapshot deletion. I assumed that objects were ordered by namespace internally/that listing a name-space would be efficient.
Is it cheaper than doing over the network? Sure. Is it cheap enough we're willing to let a single user request generate that kind of cluster IO on an unconstrained interface? Absolutely not.
Agreed, nothing wrong with doing it over the network in this case. Paul
-Greg [1]: Deleting snapshots is only feasible because every OSD maintains a sorted secondary index from snapid->set<objects>. This is only possible because snapids are issued by the monitors and clients cooperate in making sure they can't get reused after being deleted. Namespaces are generated by clients and there are no constraints on their use, reuse, or relationship to each other. We could maybe work around these problems, but it'd be building a fundamentally different interface than what namespaces currently are.
Hi Paul, I’ve turned the following idea of yours into a tracker: https://tracker.ceph.com/issues/41051 <https://tracker.ceph.com/issues/41051>
4. Common prefixes could filtered in the rgw class on the OSD instead of in radosgw
Consider a bucket with 100 folders with 1000 objects in each and only one shard
/p1/1, /p1/2, ..., /p1/1000, /p2/1, /p2/2, ..., /p2/1000, ... /p100/1000
Now a user wants to list / with aggregating common prefixes on the delimiter / and wants up to 1000 results. So there'll be 100 results returned to the client: the common prefixes p1 to p100.
How much data will be transfered between the OSDs and radosgw for this request? How many omap entries does the OSD scan?
radosgw will ask the (single) index object to list the first 1000 objects. It'll return 1000 objects in a quite unhelpful way: /p1/1, /p1/2, ...., /p1/1000
radosgw will discard 999 of these and detect one common prefix and continue the iteration at /p1/\xFF to skip the remaining entries in /p1/ if there are any. The OSD will then return everything in /p2/ in that next request and so on.
So it'll internally list every single object in that bucket. That will be a problem for large buckets and having lots of shards doesn't help either.
This shouldn't be too hard to fix: add an option "aggregate prefixes" to the RGW class method and duplicate the fast-forward logic from radosgw in cls_rgw. It doesn't even need to change the response type or anything, it just needs to limit entries in common prefixes to one result. Is this a good idea or am I missing something?
IO would be reduced by a factor of 100 for that particular pathological case. I've unfortunately seen a real-world setup that I think hits a case like that.
Eric -- J. Eric Ivancich he/him/his Red Hat Storage Ann Arbor, Michigan, USA
Right now our main focus is on the Veeam use case (VMWare backup), used with an S3 storage tier. Currently we host a bucket with 125M objects and one with 100M objects. As Paul stated, searching common prefixes can be painful. We had some cases that did not work (taking too much time, radosgw taking too much memory) until the upgrade from 14.2.1 to 14.2.2, which includes an important fix for that :-) We expect up to 400M objects per bucket. Following the 100k recommendation, we started with 4096 shards per bucket. Other cases to search common prefixes took several minutes. It helped us to reshard from 4096 to 1024, response time became nearly 3 times faster. It feels that the main reason to have shards is to get distribution of index operations' load over several PGs and therefore over several OSDs. So maybe a number of shards much higher than the number of PGs or OSDs does not help a lot? But it introduces some overhead. Maybe it would be better to have a recommendation based on the number of OSDs involved? The mentioned resharding (4096 -> 1024) itself worked ("completed successfully"), but the removal of one of the old indexes did not. The cluster saw an OSD going down, which seems to have aborted the cleanup. This OSD stayed up, but there were timeouts, probably during RocksDB compaction (from looking at the OSD log). The affected OSD has the highest number of PGs of the index pool. Again, this would suggest that a lot of shards does not help when many shards are processed together in one RocksDB. Manually removing the objects of the old index one by one was no problem. Maybe dynamic resharding could do it similarly to avoid the RocksDB overload? Or RocksDB could be made to stay responsive? Harry On 31.07.19 20:02, Paul Emmerich wrote:
Hi,
we are seeing a trend towards rather large RGW S3 buckets lately. we've worked on several clusters with 100 - 500 million objects in a single bucket, and we've been asked about the possibilities of buckets with several billion objects more than once.
From our experience: buckets with tens of million objects work just fine with no big problems usually. Buckets with hundreds of million objects require some attention. Buckets with billions of objects? "How about indexless buckets?" - "No, we need to list them".
A few stories and some questions:
1. The recommended number of objects per shard is 100k. Why? How was this default configuration derived?
It doesn't really match my experiences. We know a few clusters running with larger shards because resharding isn't possible for various reasons at the moment. They sometimes work better than buckets with lots of shards.
So we've been considering to at least double that 100k target shard size for large buckets, that would make the following point far less annoying.
2. Many shards + ordered object listing = lots of IO
Unfortunately telling people to not use ordered listings when they don't really need them doesn't really work as their software usually just doesn't support that :(
A listing request for X objects will retrieve up to X objects from each shard for ordering them. That will lead to quite a lot of traffic between the OSDs and the radosgw instances, even for relatively innocent simple queries as X defaults to 1000 usually.
Simple example: just getting the first page of a bucket listing with 4096 shards fetches around 1 GB of data from the OSD to return ~300kb or so to the S3 client.
I've got two clusters here that are only used for some relatively low-bandwidth backup use case here. However, there are a few buckets with hundreds of millions of objects that are sometimes being listed by the backup system.
The result is that this cluster has an average read IO of 1-2 GB/s, all going to the index pool. Not a big deal since that's coming from SSDs and goes over 80 Gbit/s LACP bonds. But it does pose the question about scalability as the user- visible load created by the S3 clients is quite low.
3. Deleting large buckets
Someone accidentaly put 450 million small objects into a bucket and only noticed when the cluster ran full. The bucket isn't needed, so just delete it and case closed?
Deleting is unfortunately far slower than adding objects, also radosgw-admin leaks memory during deletion: https://tracker.ceph.com/issues/40700
Increasing --max-concurrent-ios helps with deletion speed (option does effect deletion concurrency, documentation says it's only for other specific commands).
Since the deletion is going faster than new data is being added to that cluster the "solution" was to run the deletion command in a memory-limited cgroup and restart it automatically after it gets killed due to leaking.
How could the bucket deletion of the future look like? Would it be possible to put all objects in buckets into RADOS namespaces and implement some kind of efficient namespace deletion on the OSD level similar to how pool deletions are handled at a lower level?
4. Common prefixes could filtered in the rgw class on the OSD instead of in radosgw
Consider a bucket with 100 folders with 1000 objects in each and only one shard
/p1/1, /p1/2, ..., /p1/1000, /p2/1, /p2/2, ..., /p2/1000, ... /p100/1000
Now a user wants to list / with aggregating common prefixes on the delimiter / and wants up to 1000 results. So there'll be 100 results returned to the client: the common prefixes p1 to p100.
How much data will be transfered between the OSDs and radosgw for this request? How many omap entries does the OSD scan?
radosgw will ask the (single) index object to list the first 1000 objects. It'll return 1000 objects in a quite unhelpful way: /p1/1, /p1/2, ...., /p1/1000
radosgw will discard 999 of these and detect one common prefix and continue the iteration at /p1/\xFF to skip the remaining entries in /p1/ if there are any. The OSD will then return everything in /p2/ in that next request and so on.
So it'll internally list every single object in that bucket. That will be a problem for large buckets and having lots of shards doesn't help either.
This shouldn't be too hard to fix: add an option "aggregate prefixes" to the RGW class method and duplicate the fast-forward logic from radosgw in cls_rgw. It doesn't even need to change the response type or anything, it just needs to limit entries in common prefixes to one result. Is this a good idea or am I missing something?
IO would be reduced by a factor of 100 for that particular pathological case. I've unfortunately seen a real-world setup that I think hits a case like that.
Paul
On 8/2/19 3:04 AM, Harald Staub wrote:
Right now our main focus is on the Veeam use case (VMWare backup), used with an S3 storage tier. Currently we host a bucket with 125M objects and one with 100M objects.
As Paul stated, searching common prefixes can be painful. We had some cases that did not work (taking too much time, radosgw taking too much memory) until the upgrade from 14.2.1 to 14.2.2, which includes an important fix for that :-)
We expect up to 400M objects per bucket. Following the 100k recommendation, we started with 4096 shards per bucket.
Other cases to search common prefixes took several minutes. It helped us to reshard from 4096 to 1024, response time became nearly 3 times faster.
It feels that the main reason to have shards is to get distribution of index operations' load over several PGs and therefore over several OSDs. So maybe a number of shards much higher than the number of PGs or OSDs does not help a lot? But it introduces some overhead. Maybe it would be better to have a recommendation based on the number of OSDs involved?
More shards are also important for recovery. Overall recovery time for a given bucket is reduced since each shard can be recovered in parallel. With fewer shards, you get larger rados objects, each of which will take longer to recover, potentially causing a longer outage if there aren't enough copies to be active.
The mentioned resharding (4096 -> 1024) itself worked ("completed successfully"), but the removal of one of the old indexes did not. The cluster saw an OSD going down, which seems to have aborted the cleanup. This OSD stayed up, but there were timeouts, probably during RocksDB compaction (from looking at the OSD log). The affected OSD has the highest number of PGs of the index pool. Again, this would suggest that a lot of shards does not help when many shards are processed together in one RocksDB.
Manually removing the objects of the old index one by one was no problem. Maybe dynamic resharding could do it similarly to avoid the RocksDB overload? Or RocksDB could be made to stay responsive?
There's a lot of work going into making rocksdb more effective for these cases - much of it discussed in the performance weekly from July 25: https://pad.ceph.com/p/performance_weekly One of the major pieces there is sharding rocksdb into multiple column families. This reduces the size of an individual LSM-tree within rocksdb, meaning levels are smaller and compactions are faster. In testing so far this significantly reduces tail latency (60-70% reduced 99% latency for pure-omap writes, similar to an rgw bucket index workload). Josh
On 31.07.19 20:02, Paul Emmerich wrote:
Hi,
we are seeing a trend towards rather large RGW S3 buckets lately. we've worked on several clusters with 100 - 500 million objects in a single bucket, and we've been asked about the possibilities of buckets with several billion objects more than once.
From our experience: buckets with tens of million objects work just fine with no big problems usually. Buckets with hundreds of million objects require some attention. Buckets with billions of objects? "How about indexless buckets?" - "No, we need to list them".
A few stories and some questions:
1. The recommended number of objects per shard is 100k. Why? How was this default configuration derived?
It doesn't really match my experiences. We know a few clusters running with larger shards because resharding isn't possible for various reasons at the moment. They sometimes work better than buckets with lots of shards.
So we've been considering to at least double that 100k target shard size for large buckets, that would make the following point far less annoying.
2. Many shards + ordered object listing = lots of IO
Unfortunately telling people to not use ordered listings when they don't really need them doesn't really work as their software usually just doesn't support that :(
A listing request for X objects will retrieve up to X objects from each shard for ordering them. That will lead to quite a lot of traffic between the OSDs and the radosgw instances, even for relatively innocent simple queries as X defaults to 1000 usually.
Simple example: just getting the first page of a bucket listing with 4096 shards fetches around 1 GB of data from the OSD to return ~300kb or so to the S3 client.
I've got two clusters here that are only used for some relatively low-bandwidth backup use case here. However, there are a few buckets with hundreds of millions of objects that are sometimes being listed by the backup system.
The result is that this cluster has an average read IO of 1-2 GB/s, all going to the index pool. Not a big deal since that's coming from SSDs and goes over 80 Gbit/s LACP bonds. But it does pose the question about scalability as the user- visible load created by the S3 clients is quite low.
3. Deleting large buckets
Someone accidentaly put 450 million small objects into a bucket and only noticed when the cluster ran full. The bucket isn't needed, so just delete it and case closed?
Deleting is unfortunately far slower than adding objects, also radosgw-admin leaks memory during deletion: https://tracker.ceph.com/issues/40700
Increasing --max-concurrent-ios helps with deletion speed (option does effect deletion concurrency, documentation says it's only for other specific commands).
Since the deletion is going faster than new data is being added to that cluster the "solution" was to run the deletion command in a memory-limited cgroup and restart it automatically after it gets killed due to leaking.
How could the bucket deletion of the future look like? Would it be possible to put all objects in buckets into RADOS namespaces and implement some kind of efficient namespace deletion on the OSD level similar to how pool deletions are handled at a lower level?
4. Common prefixes could filtered in the rgw class on the OSD instead of in radosgw
Consider a bucket with 100 folders with 1000 objects in each and only one shard
/p1/1, /p1/2, ..., /p1/1000, /p2/1, /p2/2, ..., /p2/1000, ... /p100/1000
Now a user wants to list / with aggregating common prefixes on the delimiter / and wants up to 1000 results. So there'll be 100 results returned to the client: the common prefixes p1 to p100.
How much data will be transfered between the OSDs and radosgw for this request? How many omap entries does the OSD scan?
radosgw will ask the (single) index object to list the first 1000 objects. It'll return 1000 objects in a quite unhelpful way: /p1/1, /p1/2, ...., /p1/1000
radosgw will discard 999 of these and detect one common prefix and continue the iteration at /p1/\xFF to skip the remaining entries in /p1/ if there are any. The OSD will then return everything in /p2/ in that next request and so on.
So it'll internally list every single object in that bucket. That will be a problem for large buckets and having lots of shards doesn't help either.
This shouldn't be too hard to fix: add an option "aggregate prefixes" to the RGW class method and duplicate the fast-forward logic from radosgw in cls_rgw. It doesn't even need to change the response type or anything, it just needs to limit entries in common prefixes to one result. Is this a good idea or am I missing something?
IO would be reduced by a factor of 100 for that particular pathological case. I've unfortunately seen a real-world setup that I think hits a case like that.
Paul
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
participants (9)
-
EDH - Manuel Rios Fernandez
-
Eric Ivancich
-
Gregory Farnum
-
Harald Staub
-
J. Eric Ivancich
-
Josh Durgin
-
Lars Marowsky-Bree
-
Matthew Vernon
-
Paul Emmerich