Ceph Users Feedback Survey
Hi everyone, On behalf of the Ceph Foundation Board, I would like to announce the creation of, and cordially invite you to, the first of a recurring series of meetings focused solely on gathering feedback from the users of Ceph. The overarching goal of these meetings is to elicit feedback from the users, companies, and organizations who use Ceph in their production environments. You can find more details about the motivation behind this effort in our user survey [1] that we highly encourage all of you to take. This is an extension of the Ceph User Dev Meeting with concerted focus on Performance (led by Vincent Hsu, IBM) and Orchestration/Deployment (led by Matt Leonard, Bloomberg), to start off with. We would like to kick off this series of meetings on March 21, 2024. The survey will be open until March 18, 2024. Looking forward to hearing from you! Thanks, Neha [1] https://docs.google.com/forms/d/15aWxoG4wSQz7ziBaReVNYVv94jA0dSNQsDJGqmHCLMg/viewform?ts=65e87dd8&edit_requested=true
Hi everyone,
On behalf of the Ceph Foundation Board, I would like to announce the creation of, and cordially invite you to, the first of a recurring series of meetings focused solely on gathering feedback from the users of Ceph. The overarching goal of these meetings is to elicit feedback from the users, companies, and organizations who use Ceph in their production environments. You can find more details about the motivation behind this effort in our user survey [1] that we highly encourage all of you to take. This is an extension of the Ceph User Dev Meeting with concerted focus on Performance (led by Vincent Hsu, IBM) and Orchestration/Deployment (led by Matt Leonard, Bloomberg), to start off with. We would like to kick off this series of meetings on March 21, 2024. The survey will be open until March 18, 2024.
Looking forward to hearing from you!
Hi, Neha, we are very excited to see this opportunity to communicate with the community. Ceph have been a very important solution to our data storage requirements, however, we also see the following issues that are obstructing ceph's further extensive usage in our office: 1. We continuously observe significant performance degradation in both rbd and rgw clusters when there's pg backfilling. a): According to our investigation, this performance degradation mostly comes from the PG removals, the cost of which comes from listing the to-be-removed PGs' objects. We believe that the reason for this high-cost object listing is that bluestore stores metadata in RocksDB the read performance of which may not be superb when there are a lot of keys in it. The case is even worse in our rgw clusters, because our users tend to store small objects, the sizes of which are mostly within 200KB, in our clusters, which means data of the same total size will create much more metadata to be stored in a single OSD in the rgw clusters than the rbd clusters. This hurts even more when our clusters use EC as the data redundancy policy, because those small objects will be cut into even smaller pieces. The situation is that, if we use the default pg removal configuration, our rgw clusters can be rendered completely unavailable when we try to expand them; on the other hand, we also can't config the pg removal to be too slow most of the time, because we rely on PG Removal to spare the disk space for further usage and configuring it to be slow will result in FULL clusters. It seems that, if we can remove pg objects without listing the pg, the issue will be gone. b): It looks to us that the performance degradation also comes from the one-to-many rgw-rados object mapping. As mentioned earlier, our users tend to save small objects in our rgw clusters, and rgw saves each of these small objects as one or multiple rados objects. On the other hand, RADOS does pg backfillling in the granularity of objects. So, in our rgw clusters, PG backfilling will issue large amounts of small data reads to the underlying disks, which not only hurts the client IO performance but also slows down the backfilling itself significantly. We think maybe organizing small rgw objects into large rados objects and having a background compaction process that continuously scans those rados objects and reclaim the space of removed rgw objects will solve this issue. 2. We also have a need to store large amounts of ML training data, which in most cases are also in the form of small files; however, CephFS doesn't seem to handle this situation very well: a): we continuously see clients failing to respond to write caps reclamation requests even without data to be flushed to the cluster and the cluster not detecting this kind of clients which further lead to them not forcing the reclaiming, which can block a lot of other clients forever; in this case, our administrators often have to manually evict the client; b): since we are trying to handle ML workloads, which involves large amounts of small files, we want to use the memory space of the MDS machine sufficiently, so we usually configure the MDS memory target to 80% the total memory space. However, it seems that the MDS memory target is not a strict limit, so there are cases in which MDSes are killed by the kernel oom-killer, and if this happens, the standby-replay MDS also has a high probability of being killed by the oom-killer after it turns to active. In this case, we often observe that the MDS bootstrapping will need more memory space than its memory target to replay the journal, and there are cases that the memory is not enough and the MDSes may never come up again. Our solution to this situation is to find other machines with larger memory space to run the MDSes again. This hurts us really badly. It looks to us that the ML training is a standard write-once-read-many workload, so maybe a simple FUSE implementation based on LibRGW is good enough. The above two issues are the major obstruction of our ceph ambition at present, and we believe that we can extend the ceph usage aggressively with them solved. Thanks very much:-)
Thanks, Neha
[1] https://docs.google.com/forms/d/15aWxoG4wSQz7ziBaReVNYVv94jA0dSNQsDJGqmHCLMg/viewform?ts=65e87dd8&edit_requested=true _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Xuehan, Thank you for sharing your feedback! Those sound like good topics to discuss at the User Dev Meetings. - Neha
On Mar 13, 2024, at 11:14 AM, Xuehan Xu <xxhdx1985126@gmail.com> wrote:
Hi everyone,
On behalf of the Ceph Foundation Board, I would like to announce the creation of, and cordially invite you to, the first of a recurring series of meetings focused solely on gathering feedback from the users of Ceph. The overarching goal of these meetings is to elicit feedback from the users, companies, and organizations who use Ceph in their production environments. You can find more details about the motivation behind this effort in our user survey [1] that we highly encourage all of you to take. This is an extension of the Ceph User Dev Meeting with concerted focus on Performance (led by Vincent Hsu, IBM) and Orchestration/Deployment (led by Matt Leonard, Bloomberg), to start off with. We would like to kick off this series of meetings on March 21, 2024. The survey will be open until March 18, 2024.
Looking forward to hearing from you!
Hi, Neha, we are very excited to see this opportunity to communicate with the community.
Ceph have been a very important solution to our data storage requirements, however, we also see the following issues that are obstructing ceph's further extensive usage in our office:
1. We continuously observe significant performance degradation in both rbd and rgw clusters when there's pg backfilling. a): According to our investigation, this performance degradation mostly comes from the PG removals, the cost of which comes from listing the to-be-removed PGs' objects. We believe that the reason for this high-cost object listing is that bluestore stores metadata in RocksDB the read performance of which may not be superb when there are a lot of keys in it. The case is even worse in our rgw clusters, because our users tend to store small objects, the sizes of which are mostly within 200KB, in our clusters, which means data of the same total size will create much more metadata to be stored in a single OSD in the rgw clusters than the rbd clusters. This hurts even more when our clusters use EC as the data redundancy policy, because those small objects will be cut into even smaller pieces. The situation is that, if we use the default pg removal configuration, our rgw clusters can be rendered completely unavailable when we try to expand them; on the other hand, we also can't config the pg removal to be too slow most of the time, because we rely on PG Removal to spare the disk space for further usage and configuring it to be slow will result in FULL clusters. It seems that, if we can remove pg objects without listing the pg, the issue will be gone. b): It looks to us that the performance degradation also comes from the one-to-many rgw-rados object mapping. As mentioned earlier, our users tend to save small objects in our rgw clusters, and rgw saves each of these small objects as one or multiple rados objects. On the other hand, RADOS does pg backfillling in the granularity of objects. So, in our rgw clusters, PG backfilling will issue large amounts of small data reads to the underlying disks, which not only hurts the client IO performance but also slows down the backfilling itself significantly. We think maybe organizing small rgw objects into large rados objects and having a background compaction process that continuously scans those rados objects and reclaim the space of removed rgw objects will solve this issue. 2. We also have a need to store large amounts of ML training data, which in most cases are also in the form of small files; however, CephFS doesn't seem to handle this situation very well: a): we continuously see clients failing to respond to write caps reclamation requests even without data to be flushed to the cluster and the cluster not detecting this kind of clients which further lead to them not forcing the reclaiming, which can block a lot of other clients forever; in this case, our administrators often have to manually evict the client; b): since we are trying to handle ML workloads, which involves large amounts of small files, we want to use the memory space of the MDS machine sufficiently, so we usually configure the MDS memory target to 80% the total memory space. However, it seems that the MDS memory target is not a strict limit, so there are cases in which MDSes are killed by the kernel oom-killer, and if this happens, the standby-replay MDS also has a high probability of being killed by the oom-killer after it turns to active. In this case, we often observe that the MDS bootstrapping will need more memory space than its memory target to replay the journal, and there are cases that the memory is not enough and the MDSes may never come up again. Our solution to this situation is to find other machines with larger memory space to run the MDSes again. This hurts us really badly. It looks to us that the ML training is a standard write-once-read-many workload, so maybe a simple FUSE implementation based on LibRGW is good enough.
The above two issues are the major obstruction of our ceph ambition at present, and we believe that we can extend the ceph usage aggressively with them solved.
Thanks very much:-)
Thanks, Neha
[1] https://docs.google.com/forms/d/15aWxoG4wSQz7ziBaReVNYVv94jA0dSNQsDJGqmHCLMg/viewform?ts=65e87dd8&edit_requested=true _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi, Neha, we are very excited to see this opportunity to communicate with the community.
Ceph have been a very important solution to our data storage requirements, however, we also see the following issues that are obstructing ceph's further extensive usage in our office:
1. We continuously observe significant performance degradation in both rbd and rgw clusters when there's pg backfilling. a): According to our investigation, this performance degradation mostly comes from the PG removals, the cost of which comes from listing the to-be-removed PGs' objects. We believe that the reason for this high-cost object listing is that bluestore stores metadata in RocksDB the read performance of which may not be superb when there are a lot of keys in it. The case is even worse in our rgw clusters, because our users tend to store small objects, the sizes of which are mostly within 200KB, in our clusters, which means data of the same total size will create much more metadata to be stored in a single OSD in the rgw clusters than the rbd clusters.
This is very much a known phenomenon. Fortunately my own ML workload uses ~50MB objects. Is your RGW index pool on NVMe SSDs? How many of them, with how many PGs? Spreading the index pool across more drives and increasing the PG count above the formal guidance may help. And is your bucket pool on SSDs?
from the one-to-many rgw-rados object mapping. As mentioned earlier, our users tend to save small objects in our rgw clusters, and rgw saves each of these small objects as one or multiple rados objects. On the other hand, RADOS does pg backfillling in the granularity of objects. So, in our rgw clusters, PG backfilling will issue large amounts of small data reads to the underlying disks, which not only hurts the client IO performance but also slows down the backfilling itself significantly. We think maybe organizing small rgw objects into large rados objects and having a background compaction process that continuously scans those rados objects and reclaim the space of removed rgw objects will solve this issue.
This is sometimes referred to as packing small objects -- the hole / compaction issue is an obvious wrinkle. I was told a few years ago that certain folks had a prototype here, but I haven't seen a PR.
b): since we are trying to handle ML workloads, which involves large amounts of small files, we want to use the memory space of the MDS machine sufficiently, so we usually configure the MDS memory target to 80% the total memory space. However, it seems that the MDS memory target is not a strict limit, so there are cases in which MDSes are killed by the kernel oom-killer, and if this happens, the standby-replay MDS also has a high probability of being killed by the oom-killer after it turns to active. In this case, we often observe that the MDS bootstrapping will need more memory space than its memory target to replay the journal, and there are cases that the memory is not enough and the MDSes may never come up again. Our solution to this situation is to find other machines with larger memory space to run the MDSes again. This hurts us really badly.
Do you have multiple MDS with pinning?
It looks to us that the ML training is a standard write-once-read-many workload, so maybe a simple FUSE implementation based on LibRGW is good enough.
The above two issues are the major obstruction of our ceph ambition at present, and we believe that we can extend the ceph usage aggressively with them solved.
Thanks very much:-)
Thanks, Neha
[1] https://docs.google.com/forms/d/15aWxoG4wSQz7ziBaReVNYVv94jA0dSNQsDJGqmHCLMg/viewform?ts=65e87dd8&edit_requested=true _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On 5/18/24 07:35, Anthony D'Atri wrote:
Hi, Neha, we are very excited to see this opportunity to communicate with the community.
Ceph have been a very important solution to our data storage requirements, however, we also see the following issues that are obstructing ceph's further extensive usage in our office:
1. We continuously observe significant performance degradation in both rbd and rgw clusters when there's pg backfilling. a): According to our investigation, this performance degradation mostly comes from the PG removals, the cost of which comes from listing the to-be-removed PGs' objects. We believe that the reason for this high-cost object listing is that bluestore stores metadata in RocksDB the read performance of which may not be superb when there are a lot of keys in it. The case is even worse in our rgw clusters, because our users tend to store small objects, the sizes of which are mostly within 200KB, in our clusters, which means data of the same total size will create much more metadata to be stored in a single OSD in the rgw clusters than the rbd clusters. This is very much a known phenomenon. Fortunately my own ML workload uses ~50MB objects.
Is your RGW index pool on NVMe SSDs? How many of them, with how many PGs? Spreading the index pool across more drives and increasing the PG count above the formal guidance may help.
And is your bucket pool on SSDs?
I would be very interested to hear if the compact-on-deletion feature helps in this case. Here's the commit where it was added: https://github.com/ceph/ceph/pull/47221/commits/fba5488728e89d9b0a1c1ab94b70... The idea here is that when you are in a situation that involves interleaved lists and deletes without a lot of writes, you will end up repeatedly iterating over the tomstones for deleted entries. If you don't have a concurrent write workload (say for OSDs that are having data migrated away from them), you can end up making listing extremely expensive because there are no compactions cleaning the tombstones up. This feature makes it so that if RocksDB encounters too many tombstones during iteration, it will automatically trigger a compaction. You'll need the latest versions of Pacific/Quincy or reef to be able to utilize/tune this feature, but we've seen it make a huge difference in several large production clusters.
from the one-to-many rgw-rados object mapping. As mentioned earlier, our users tend to save small objects in our rgw clusters, and rgw saves each of these small objects as one or multiple rados objects. On the other hand, RADOS does pg backfillling in the granularity of objects. So, in our rgw clusters, PG backfilling will issue large amounts of small data reads to the underlying disks, which not only hurts the client IO performance but also slows down the backfilling itself significantly. We think maybe organizing small rgw objects into large rados objects and having a background compaction process that continuously scans those rados objects and reclaim the space of removed rgw objects will solve this issue. This is sometimes referred to as packing small objects -- the hole / compaction issue is an obvious wrinkle. I was told a few years ago that certain folks had a prototype here, but I haven't seen a PR.
b): since we are trying to handle ML workloads, which involves large amounts of small files, we want to use the memory space of the MDS machine sufficiently, so we usually configure the MDS memory target to 80% the total memory space. However, it seems that the MDS memory target is not a strict limit, so there are cases in which MDSes are killed by the kernel oom-killer, and if this happens, the standby-replay MDS also has a high probability of being killed by the oom-killer after it turns to active. In this case, we often observe that the MDS bootstrapping will need more memory space than its memory target to replay the journal, and there are cases that the memory is not enough and the MDSes may never come up again. Our solution to this situation is to find other machines with larger memory space to run the MDSes again. This hurts us really badly. Do you have multiple MDS with pinning?
That was my thought too. Once a single MDS can't keep up with trimming you can try to make it more aggressive, but it's also probably time to start thinking about going multi-active and using pinning (ephemeral or otherwise) to distribute both memory and trimming across multiple MDSes. I suspect one of the problems here is that we keep iterating over items in the cache that we can't evict (we used to do this in bluestore as well), but I haven't looked at it closely. At some point I'd really like to dig into the MDS cache and see if there's anything we can do to make this better.
It looks to us that the ML training is a standard write-once-read-many workload, so maybe a simple FUSE implementation based on LibRGW is good enough.
ML training (at least for something like ResNet50) is all about ingestion images fast enough to keep the GPUs busy. For H100s it's going to be about a 1GB/s per card give or take. LibCephFS itself is actually pretty good for this kind of workload (and beats the kernel client!). FUSE is a problem though. In general, we need to figure out how to reduce the amount of work to get data from the network to the GPU. (Even if it's not quite as fancy as something like GPUDirect).
The above two issues are the major obstruction of our ceph ambition at present, and we believe that we can extend the ceph usage aggressively with them solved.
Thanks very much:-)
Thanks, Neha
[1] https://docs.google.com/forms/d/15aWxoG4wSQz7ziBaReVNYVv94jA0dSNQsDJGqmHCLMg/viewform?ts=65e87dd8&edit_requested=true _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Best Regards, Mark Nelson Head of Research and Development Clyso GmbH p: +49 89 21552391 12 | a: Minnesota, USA w: https://clyso.com | e: mark.nelson@clyso.com We are hiring: https://www.clyso.com/jobs/
Hi everyone, At the next Ceph User + Dev Monthly Meetup (Wednesday, April 17, 7:00 – 8:00 am PT), we'll go over the results of the Ceph Users Feedback Survey that we did last month. If you missed taking the survey, feel free to go over the questions attached in the email below. We look forward to your participation in order to make this discussion more meaningful. Thanks, Neha On Tue, Mar 12, 2024 at 9:00 AM Neha Ojha <nojha@redhat.com> wrote:
Hi everyone,
On behalf of the Ceph Foundation Board, I would like to announce the creation of, and cordially invite you to, the first of a recurring series of meetings focused solely on gathering feedback from the users of Ceph. The overarching goal of these meetings is to elicit feedback from the users, companies, and organizations who use Ceph in their production environments. You can find more details about the motivation behind this effort in our user survey [1] that we highly encourage all of you to take. This is an extension of the Ceph User Dev Meeting with concerted focus on Performance (led by Vincent Hsu, IBM) and Orchestration/Deployment (led by Matt Leonard, Bloomberg), to start off with. We would like to kick off this series of meetings on March 21, 2024. The survey will be open until March 18, 2024.
Looking forward to hearing from you!
Thanks, Neha
participants (4)
-
Anthony D'Atri
-
Mark Nelson
-
Neha Ojha
-
Xuehan Xu