Does dynamic resharding block I/Os by design?
Hi, I have a Ceph cluster used for RGW and RBD. I found that all I/Os to RGW seemed to be blocked while dynamic resharding. Could you tell me whether this behavior is by design or not? I attached a graph which means I/O seemed to be blocked. Here x-axis is time and y-axis is the number of RADOS objects. In addition, dynamic resharding was run between 16:22:30 and 16:31:30. I read the official documents about dynamic resharding. But there is no description about blocking during dynamic resharding. https://docs.ceph.com/en/octopus/radosgw/dynamicresharding/ In addition, I read the following Red Hat's blog post. https://www.redhat.com/ja/blog/ceph-rgw-dynamic-bucket-sharding-performance-...
You do not need to stop reading or writing objects to the bucket while resharding is happening.
It would mean the dynamic resharding is online operation. However, it's not clear whether this feature blocks I/Os or not. Thanks, Satoru
2021年5月18日(火) 9:23 Satoru Takeuchi <satoru.takeuchi@gmail.com>:
Hi,
I have a Ceph cluster used for RGW and RBD. I found that all I/Os to RGW seemed to be blocked while dynamic resharding. Could you tell me whether this behavior is by design or not?
I attached a graph which means I/O seemed to be blocked. Here x-axis is time and y-axis is the number of RADOS objects. In addition, dynamic resharding was run between 16:22:30 and 16:31:30.
I uploaded my image to google photos (thank you Anthony to point it out). https://photos.app.goo.gl/ZFJg7CQ4z6gfqmSX6
I read the official documents about dynamic resharding. But there is no description about blocking during dynamic resharding.
https://docs.ceph.com/en/octopus/radosgw/dynamicresharding/
In addition, I read the following Red Hat's blog post.
https://www.redhat.com/ja/blog/ceph-rgw-dynamic-bucket-sharding-performance-...
You do not need to stop reading or writing objects to the bucket while resharding is happening.
It would mean the dynamic resharding is online operation. However, it's not clear whether this feature blocks I/Os or not.
Thanks, Satoru
2021年5月18日(火) 14:09 Satoru Takeuchi <satoru.takeuchi@gmail.com>:
2021年5月18日(火) 9:23 Satoru Takeuchi <satoru.takeuchi@gmail.com>:
Hi,
I have a Ceph cluster used for RGW and RBD. I found that all I/Os to RGW seemed to be blocked while dynamic resharding. Could you tell me whether this behavior is by design or not?
I attached a graph which means I/O seemed to be blocked. Here x-axis is time and y-axis is the number of RADOS objects. In addition, dynamic resharding was run between 16:22:30 and 16:31:30.
I uploaded my image to google photos (thank you Anthony to point it out).
Can someone please answer this question? I can show you more detailed information if desired.
https://photos.app.goo.gl/ZFJg7CQ4z6gfqmSX6
I read the official documents about dynamic resharding. But there is no description about blocking during dynamic resharding.
https://docs.ceph.com/en/octopus/radosgw/dynamicresharding/
In addition, I read the following Red Hat's blog post.
https://www.redhat.com/ja/blog/ceph-rgw-dynamic-bucket-sharding-performance-...
You do not need to stop reading or writing objects to the bucket while
resharding is happening.
It would mean the dynamic resharding is online operation. However, it's not clear whether this feature blocks I/Os or not.
Thanks, Satoru
2021年5月21日(金) 12:26 Satoru Takeuchi <satoru.takeuchi@gmail.com>:
2021年5月18日(火) 14:09 Satoru Takeuchi <satoru.takeuchi@gmail.com>:
2021年5月18日(火) 9:23 Satoru Takeuchi <satoru.takeuchi@gmail.com>:
Hi,
I have a Ceph cluster used for RGW and RBD. I found that all I/Os to RGW seemed to be blocked while dynamic resharding. Could you tell me whether this behavior is by design or not?
I attached a graph which means I/O seemed to be blocked. Here x-axis is time and y-axis is the number of RADOS objects. In addition, dynamic resharding was run between 16:22:30 and 16:31:30.
I uploaded my image to google photos (thank you Anthony to point it out).
Can someone please answer this question? I can show you more detailed information if desired.
I submitted an issue for this problem. https://tracker.ceph.com/issues/50945 Thanks, Satoru
https://photos.app.goo.gl/ZFJg7CQ4z6gfqmSX6
I read the official documents about dynamic resharding. But there is no description about blocking during dynamic resharding.
https://docs.ceph.com/en/octopus/radosgw/dynamicresharding/
In addition, I read the following Red Hat's blog post.
https://www.redhat.com/ja/blog/ceph-rgw-dynamic-bucket-sharding-performance-...
You do not need to stop reading or writing objects to the bucket while resharding is happening.
It would mean the dynamic resharding is online operation. However, it's not clear whether this feature blocks I/Os or not.
Thanks, Satoru
Hi Satoru, Writes to a bucket are blocked while it is resharding, reads aren't. Dynamic bucket resharding just means that it's automatically scheduled for resharding once it reaches ~100k objects/shard There were some discussions around non-blocking resharding operations (https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/NG56XXAM5A4JONT4BGP...) but I don't know what Ceph release this is planned for. On Sun, May 23, 2021 at 9:23 PM Satoru Takeuchi <satoru.takeuchi@gmail.com> wrote:
2021年5月21日(金) 12:26 Satoru Takeuchi <satoru.takeuchi@gmail.com>:
2021年5月18日(火) 14:09 Satoru Takeuchi <satoru.takeuchi@gmail.com>:
2021年5月18日(火) 9:23 Satoru Takeuchi <satoru.takeuchi@gmail.com>:
Hi,
I have a Ceph cluster used for RGW and RBD. I found that all I/Os to RGW seemed to be blocked while dynamic resharding. Could you tell me whether this behavior is by design or not?
I attached a graph which means I/O seemed to be blocked. Here x-axis is time and y-axis is the number of RADOS objects. In addition, dynamic resharding was run between 16:22:30 and 16:31:30.
I uploaded my image to google photos (thank you Anthony to point it out).
Can someone please answer this question? I can show you more detailed information if desired.
I submitted an issue for this problem.
https://tracker.ceph.com/issues/50945
Thanks, Satoru
https://photos.app.goo.gl/ZFJg7CQ4z6gfqmSX6
I read the official documents about dynamic resharding. But there is no description about blocking during dynamic resharding.
https://docs.ceph.com/en/octopus/radosgw/dynamicresharding/
In addition, I read the following Red Hat's blog post.
https://www.redhat.com/ja/blog/ceph-rgw-dynamic-bucket-sharding-performance-...
You do not need to stop reading or writing objects to the bucket while resharding is happening.
It would mean the dynamic resharding is online operation. However, it's not clear whether this feature blocks I/Os or not.
Thanks, Satoru
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Alexandre Marangone
-
Satoru Takeuchi