Veeam backups to radosgw seem to be very slow
We have a customer that tries to use veeam with our rgw objectstorage and it seems to be blazingly slow. What also seems to be strange, that veeam sometimes show "bucket does not exist" or "permission denied". I've tested parallel and everything seems to work fine from the s3cmd/aws cli standpoint. Does anyone here ever experienced veeam problems with rgw? Cheers Boris
Den tis 25 apr. 2023 kl 15:02 skrev Boris Behrens <bb@kervyn.de>:
We have a customer that tries to use veeam with our rgw objectstorage and it seems to be blazingly slow. What also seems to be strange, that veeam sometimes show "bucket does not exist" or "permission denied". I've tested parallel and everything seems to work fine from the s3cmd/aws cli standpoint. Does anyone here ever experienced veeam problems with rgw?
Mostly the issue that if you do not set block sizes to a large value, it will create REALLY small files. If "problems" relates to "poor MB/s perf", then this could be it. TCP will not get up to any speed if you keep the S3 objects to very small sizes. Look at https://community.veeam.com/blogs-and-podcasts-57/sobr-veeam-capacity-tier-c... for "extra large blocks" to make them 8M at least. We had one Veeam installation vomit millions of files onto our rgw-S3 at an average size of 180k per object, and at those sizes, you will see very poor throughput and the many objs/MB will hurt all other kinds of performance like listing the bucket and so on. -- May the most significant bit of your life be positive.
We have a customer that tries to use veeam with our rgw objectstorage and it seems to be blazingly slow. What also seems to be strange, that veeam sometimes show "bucket does not exist" or "permission denied". I've tested parallel and everything seems to work fine from the s3cmd/aws cli standpoint. Does anyone here ever experienced veeam problems with rgw?
Mostly the issue that if you do not set block sizes to a large value, it will create REALLY small files. If "problems" relates to "poor MB/s perf", then this could be it. TCP will not get up to any speed if you keep the S3 objects to very small sizes.
Look at https://community.veeam.com/blogs-and-podcasts-57/sobr-veeam-capacity-tier-c... for "extra large blocks" to make them 8M at least. We had one Veeam installation vomit millions of files onto our rgw-S3 at an average size of 180k per object, and at those sizes, you will see very poor throughput and the many objs/MB will hurt all other kinds of performance like listing the bucket and so on.
Depending on the bucket pool and the size distribution, there could be significant space amp too. A while back I looked at a cluster that was using Ceph's iscsi-gw as a destination for Veeam, they were experiencing very high latency, on the order of 100ms. I wonder if a variant of this was in play.
Hi, I’ve tested that combination once last year. My experience was similar. It was dead-slow. But if I remember correctly my conclusion was that Veeam was sending very slowly lots of rather small objects without any parallelism. But apart from the cruel slowness I didn’t have problems of the “bucket does not exist”/“permission denied” type. I don’t have the setup anymore. But I think it might be worth checking what kind of objects Veeam puts into its buckets. Ciao, Uli
On 25. Apr 2023, at 15:01, Boris Behrens <bb@kervyn.de> wrote:
We have a customer that tries to use veeam with our rgw objectstorage and it seems to be blazingly slow.
What also seems to be strange, that veeam sometimes show "bucket does not exist" or "permission denied".
I've tested parallel and everything seems to work fine from the s3cmd/aws cli standpoint.
Does anyone here ever experienced veeam problems with rgw?
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
"bucket does not exist" or "permission denied". Had received similar error messages with another client program. The default region did not match the region of the cluster. ___________________________________ ceph ambassador DACH ceph consultant since 2012 Clyso GmbH - Premier Ceph Foundation Member https://www.clyso.com/ Am 25.04.23 um 15:01 schrieb Boris Behrens:
We have a customer that tries to use veeam with our rgw objectstorage and it seems to be blazingly slow.
What also seems to be strange, that veeam sometimes show "bucket does not exist" or "permission denied".
I've tested parallel and everything seems to work fine from the s3cmd/aws cli standpoint.
Does anyone here ever experienced veeam problems with rgw?
Cheers Boris _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Janne, I will hand that to the customer.
Look at https://community.veeam.com/blogs-and-podcasts-57/sobr-veeam -capacity-tier-calculations-and-considerations-in-v11-2548 for "extra large blocks" to make them 8M at least. We had one Veeam installation vomit millions of files onto our rgw-S3 at an average size of 180k per object, and at those sizes, you will see very poor throughput and the many objs/MB will hurt all other kinds of performance like listing the bucket and so on.
@joachim What do you mean with "default region"? I just checked the period and it aligns. I've told them to try to get more information from it "bucket does not exist" or "permission denied".
Had received similar error messages with another client program. The default region did not match the region of the cluster.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
participants (5)
-
Anthony D'Atri
-
Boris Behrens
-
Janne Johansson
-
Joachim Kraftmayer - ceph ambassador
-
Ulrich Klein