Hi, Neha, we are very excited to see this opportunity to communicate with the community. Ceph have been a very important solution to our data storage requirements, however, we also see the following issues that are obstructing ceph's further extensive usage in our office: 1. We continuously observe significant performance degradation in both rbd and rgw clusters when there's pg backfilling. a): According to our investigation, this performance degradation mostly comes from the PG removals, the cost of which comes from listing the to-be-removed PGs' objects. We believe that the reason for this high-cost object listing is that bluestore stores metadata in RocksDB the read performance of which may not be superb when there are a lot of keys in it. The case is even worse in our rgw clusters, because our users tend to store small objects, the sizes of which are mostly within 200KB, in our clusters, which means data of the same total size will create much more metadata to be stored in a single OSD in the rgw clusters than the rbd clusters. This hurts even more when our clusters use EC as the data redundancy policy, because those small objects will be cut into even smaller pieces. The situation is that, if we use the default pg removal configuration, our rgw clusters can be rendered completely unavailable when we try to expand them; on the other hand, we also can't config the pg removal to be too slow most of the time, because we rely on PG Removal to spare the disk space for further usage and configuring it to be slow will result in FULL clusters. It seems that, if we can remove pg objects without listing the pg, the issue will be gone. b): It looks to us that the performance degradation also comes from the one-to-many rgw-rados object mapping. As mentioned earlier, our users tend to save small objects in our rgw clusters, and rgw saves each of these small objects as one or multiple rados objects. On the other hand, RADOS does pg backfillling in the granularity of objects. So, in our rgw clusters, PG backfilling will issue large amounts of small data reads to the underlying disks, which not only hurts the client IO performance but also slows down the backfilling itself significantly. We think maybe organizing small rgw objects into large rados objects and having a background compaction process that continuously scans those rados objects and reclaim the space of removed rgw objects will solve this issue. 2. We also have a need to store large amounts of ML training data, which in most cases are also in the form of small files; however, CephFS doesn't seem to handle this situation very well: a): we continuously see clients failing to respond to write caps reclamation requests even without data to be flushed to the cluster and the cluster not detecting this kind of clients which further lead to them not forcing the reclaiming, which can block a lot of other clients forever; in this case, our administrators often have to manually evict the client; b): since we are trying to handle ML workloads, which involves large amounts of small files, we want to use the memory space of the MDS machine sufficiently, so we usually configure the MDS memory target to 80% the total memory space. However, it seems that the MDS memory target is not a strict limit, so there are cases in which MDSes are killed by the kernel oom-killer, and if this happens, the standby-replay MDS also has a high probability of being killed by the oom-killer after it turns to active. In this case, we often observe that the MDS bootstrapping will need more memory space than its memory target to replay the journal, and there are cases that the memory is not enough and the MDSes may never come up again. Our solution to this situation is to find other machines with larger memory space to run the MDSes again. This hurts us really badly. It looks to us that the ML training is a standard write-once-read-many workload, so maybe a simple FUSE implementation based on LibRGW is good enough. The above two issues are the major obstruction of our ceph ambition at present, and we believe that we can extend the ceph usage aggressively with them solved. Thanks very much:-) > > Thanks, > Neha > > [1] https://docs.google.com/forms/d/15aWxoG4wSQz7ziBaReVNYVv94jA0dSNQsDJGqmHCLM… > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

Hi everyone, At the next Ceph User + Dev Monthly Meetup (Wednesday, April 17, 7:00 – 8:00 am PT), we'll go over the results of the Ceph Users Feedback Survey that we did last month. If you missed taking the survey, feel free to go over the questions attached in the email below. We look forward to your participation in order to make this discussion more meaningful. Thanks, Neha On Tue, Mar 12, 2024 at 9:00 AM Neha Ojha <nojha(a)redhat.com> wrote:

This is very much a known phenomenon. Fortunately my own ML workload uses ~50MB objects. Is your RGW index pool on NVMe SSDs? How many of them, with how many PGs? Spreading the index pool across more drives and increasing the PG count above the formal guidance may help. And is your bucket pool on SSDs? This is sometimes referred to as packing small objects -- the hole / compaction issue is an obvious wrinkle. I was told a few years ago that certain folks had a prototype here, but I haven't seen a PR. Do you have multiple MDS with pinning?