Hello Anthony, Sorry for the late reply. My thought process behind it was that maybe there's some kind of indexing that Ceph does under the hood, and perhaps the bucket structure could influence that. But if you say it's not the case, then I was on the wrong path. Sorry for the daley, but I also wanted to gather info.
How many millions?
About 75 millions.
How big are they?
They vary from ~500kb to a couple of megabytes, say 5mb. I wouldn't be able to tell you if most files are closer to 5mb or to 500kb though, but if that's important I can try to figure it out.
Are you writing them to a single bucket?
Yes. All these files are in a single bucket.
How is the index pool configured? On what media? Same with the bucket pool.
I wouldn't be able to answer that unfortunately.
Which Ceph release?
Pacific (https://docs.ceph.com/en/pacific/).
Sharding config? Are you mixing in bucket list operations ?
Do you have the ability to utilize more than one bucket? If you can limit
We don't use list operations on this bucket, but the Ceph infrastructure is shared across multiple companies and we are aware that there are others using list operations *on other buckets*. But also, I can say that list operations in this bucket IIRC are failing (to a point where we don't have the exact metric of how many objects are in the bucket). The provider has a prometheus exporter which fails to expert the metrics in production currently. the number of objects in a bucket that might help. Technically it should be possible, but I'd assume that Ceph can abstract this complexity for the bucket user so that we don't have to care for that. If we do it, I would see it as a workaround more than a real solution.
If your application keeps track of object names you might try indexless buckets.
I didn't know there was this possibility. I don't know how Ceph works under the hood, but assuming that all files are ultimately written to the same folder in disk, could that be a problem? I have faced in the past struggle with linux file system getting too slow due to too many files written to the same folder. Thanks for the help already! Best regards, *Renann Prado* On Sat, Feb 3, 2024 at 7:13 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
The slashes don’t mean much if anything to Ceph. Buckets are not hierarchical filesystems.
You speak of millions of files. How many millions?
How big are they? Very small objects stress any object system. Very large objects may be multi part uploads that stage to slow media or otherwise add overhead.
Are you writing them to a single bucket?
How is the index pool configured? On what media? Same with the bucket pool.
Which Ceph release? Sharding config? Are you mixing in bucket list operations ?
It could be that you have an older release or a cluster set up on an older release that doesn’t effectively auto-reshard the bucket index. If the index pool is set up poorly - slow media, too few OSDs, too few PGs - that may contribute.
In some circumstances pre-sharding might help.
Do you have the ability to utilize more than one bucket? If you can limit the number of objects in a bucket that might help.
If your application keeps track of object names you might try indexless buckets.
On Feb 3, 2024, at 12:57 PM, Renann Prado <prado.renann@gmail.com> wrote:
Hello,
I have an issue at my company where we have an underperforming Ceph instance. The issue that we have is that sometimes writing files to Ceph via S3 API (our only option) takes up to 40s, which is too long for us. We are a bit limited on what we can do to investigate why it's performing so badly, because we have a service provider in between, so getting to the bottom of this really is not that easy.
That being said, the way we use the S3 APi (again, Ceph under the hood) is by writing all files (multiple millions) to the root, so we don't use *no* folder-like structure e.g. we write */<uuid>* instead of */this/that/<uuid>* .
The question is:
Does anybody know whether Ceph has performance gains when you create a folder structure vs when you don't? Looking at Ceph's documentation I could not find such information.
Best regards,
*Renann Prado* _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io