RGW - tracking new bucket creation and bucket usage
Hello I’m looking for suggestions how to track bucket creation over s3 api and bucket usage (num of objects and size) of all buckets in time. In our RGW setup, we have a custom client panel, where like 85% percent of buckets are created which is easy for us to then track the newly created buckets and assigned them to our clients, however we also enable them to create the buckets using the s3 api and there the tracking bucket to client is harder. We also want to periodically track the bucket usage stats (num of objects and size) of all buckets in time. Now we are using the GetBucketInfo from the AdminOPS api - https://docs.ceph.com/en/quincy/radosgw/adminops/#id44 with the stats=true option GET /admin/bucket?stats=1 which returns all buckets with the number of objects and size we then parse. We also use it for the tracking of newly created buckets as it gives you all of the buckets with their owners at one place, but we are not using the metadata in that case. This however in our environment, where we have close to ~400 buckets isn’t scalable and the calls can take from 10 to 400 to even 6000s to complete. So I’m looking for suggestions on how to get this data in time better/smarter. I know there is a bunch of prometheus exporters like https://github.com/pcuzner/rgw-exporter but they are usually just using the same AdminOPS api so that doesn’t seem like a better solution. I’m also working on sending all of our RGW ops logs to elastic which would give me the ability to easily query the CreateBucket calls, but that only solves the bucket creation and not the usage stats. How would you handle this? Do you have any suggestions? Thanks a lot. Ondrej
Now we are using the GetBucketInfo from the AdminOPS api - https://docs.ceph.com/en/quincy/radosgw/adminops/#id44 with the stats=true option GET /admin/bucket?stats=1 which returns all buckets with the number of objects and size we then parse. We also use it for the tracking of newly created buckets as it gives you all of the buckets with their owners at one place, but we are not using the metadata in that case.
This however in our environment, where we have close to ~400 buckets isn’t scalable and the calls can take from 10 to 400 to even 6000s to complete.
Can you time how long it takes for you on a mon-or-other-host-with-admin-keys to run radosgw-admin bucket list and then loop over those and run radosgw-admin bucket stats on each? That would be the bottom time limit in my view, for a query that has no prior knowledge. If this is (lots) faster than the api, then you could just do that somewhere and publish the results internally for your use. -- May the most significant bit of your life be positive.
Hi,
On 15 Mar 2024, at 01:07, Ondřej Kukla <ondrej@kuuk.la> wrote:
Hello I’m looking for suggestions how to track bucket creation over s3 api and bucket usage (num of objects and size) of all buckets in time.
In our RGW setup, we have a custom client panel, where like 85% percent of buckets are created which is easy for us to then track the newly created buckets and assigned them to our clients, however we also enable them to create the buckets using the s3 api and there the tracking bucket to client is harder. We also want to periodically track the bucket usage stats (num of objects and size) of all buckets in time.
Now we are using the GetBucketInfo from the AdminOPS api - https://docs.ceph.com/en/quincy/radosgw/adminops/#id44 with the stats=true option GET /admin/bucket?stats=1 which returns all buckets with the number of objects and size we then parse. We also use it for the tracking of newly created buckets as it gives you all of the buckets with their owners at one place, but we are not using the metadata in that case.
This however in our environment, where we have close to ~400 buckets isn’t scalable and the calls can take from 10 to 400 to even 6000s to complete.
So I’m looking for suggestions on how to get this data in time better/smarter. I know there is a bunch of prometheus exporters like https://github.com/pcuzner/rgw-exporter but they are usually just using the same AdminOPS api so that doesn’t seem like a better solution.
I’m also working on sending all of our RGW ops logs to elastic which would give me the ability to easily query the CreateBucket calls, but that only solves the bucket creation and not the usage stats.
How would you handle this? Do you have any suggestions?
Do you try this [1] exporter? May be this code performance is better for you? k [1] https://github.com/blemmenes/radosgw_usage_exporter
participants (3)
-
Janne Johansson
-
Konstantin Shalygin
-
Ondřej Kukla