Hi all, Is there any configuration to slow down keys/s in recovery mode? Thanks.
On 12/1/20 12:37 AM, Seena Fallah wrote:
Hi all,
Is there any configuration to slow down keys/s in recovery mode?
Not just keys, but you can limit recovery / backfill like this: ceph tell 'osd.*' injectargs '--osd_max_backfills 1' ceph tell 'osd.*' injectargs '--osd_recovery_max_active 1' Gr. Stefan
I did the same but it moved 200K keys/s! On Wed, Dec 2, 2020 at 5:14 PM Stefan Kooman <stefan@bit.nl> wrote:
On 12/1/20 12:37 AM, Seena Fallah wrote:
Hi all,
Is there any configuration to slow down keys/s in recovery mode?
Not just keys, but you can limit recovery / backfill like this:
ceph tell 'osd.*' injectargs '--osd_max_backfills 1' ceph tell 'osd.*' injectargs '--osd_recovery_max_active 1'
Gr. Stefan
This is what I used in recovery: osd max backfills = 1 osd recovery max active = 1 osd recovery op priority = 1 osd recovery priority = 1 osd recovery sleep ssd = 0.2 But it doesn't help much! On Wed, Dec 2, 2020 at 5:23 PM Stefan Kooman <stefan@bit.nl> wrote:
On 12/2/20 2:46 PM, Seena Fallah wrote:
I did the same but it moved 200K keys/s! You might also want to decrease the op priority (as in _increasing_ the number) of "osd_recovery_op_priority".
Gr. Stefan
On 12/2/20 2:55 PM, Seena Fallah wrote:
This is what I used in recovery: osd max backfills = 1 osd recovery max active = 1 osd recovery op priority = 1
^^ Shouldn't this go to 63 instead of 1? At least if I read this post from SUSE correctly I think it should [1].
osd recovery priority = 1
^^ same for this.
osd recovery sleep ssd = 0.2
Gr. Stefan [1]: https://www.suse.com/support/kb/doc/?id=000019693
I don't think so! I want to slow down the recovery not speed up and it says I should reduce these values. On Wed, Dec 2, 2020 at 5:31 PM Stefan Kooman <stefan@bit.nl> wrote:
On 12/2/20 2:55 PM, Seena Fallah wrote:
This is what I used in recovery: osd max backfills = 1 osd recovery max active = 1 osd recovery op priority = 1
^^ Shouldn't this go to 63 instead of 1? At least if I read this post from SUSE correctly I think it should [1].
osd recovery priority = 1
^^ same for this.
osd recovery sleep ssd = 0.2
Gr. Stefan
On 12/2/20 3:04 PM, Seena Fallah wrote:
I don't think so! I want to slow down the recovery not speed up and it says I should reduce these values.
osd recovery op priority: This is the priority set for recovery operation. Lower the number, higher the recovery priority. Higher recovery priority might cause performance degradation until recovery completes. "Lower the number, higher the recovery priority." With that reasoning "higher the number, lower the recovery priority" should also be true. I would try "63" for both priorities. Gr. Stefan
Am 02.12.20 um 15:04 schrieb Seena Fallah:
I don't think so! I want to slow down the recovery not speed up and it says I should reduce these values.
I read the documentation the same. Low value = low weight, High value = high weight. [1] Operations with higher weight get easier dispatched. May I ask which exact issue are you seeing and which ceph version are you using? Peter [1] https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html/...
[1] https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html/... Can you please point me to the correct section. I don't find the point of
"Higher recovery priority might cause performance degradation until recovery completes." But what about this statement? I found this that it means if I set priority to 63, I will lose the cluster performance for clients. Am I wrong? Does it mean the performance for recovery? I'm using nautilus 14.2.14. this doc. On Wed, Dec 2, 2020 at 7:04 PM Peter Lieven <pl@kamp.de> wrote:
Am 02.12.20 um 15:04 schrieb Seena Fallah:
I don't think so! I want to slow down the recovery not speed up and it says I should reduce these values.
I read the documentation the same. Low value = low weight, High value = high weight. [1]
Operations with higher weight get easier dispatched.
May I ask which exact issue are you seeing and which ceph version are you using?
Peter
[1] https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html/...
If it uses PriorityQueue <https://en.wikipedia.org/wiki/Priority_queue#:~:text=In%20computer%20science%2C%20a%20priority,an%20element%20with%20low%20priority.> Data Structure an element with high priority should be dequeued before an element with low priority. On Wed, Dec 2, 2020 at 7:32 PM Seena Fallah <seenafallah@gmail.com> wrote:
"Higher recovery priority might cause performance degradation until recovery completes." But what about this statement? I found this that it means if I set priority to 63, I will lose the cluster performance for clients. Am I wrong? Does it mean the performance for recovery?
[1] https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html/... Can you please point me to the correct section. I don't find the point of
I'm using nautilus 14.2.14. this doc.
On Wed, Dec 2, 2020 at 7:04 PM Peter Lieven <pl@kamp.de> wrote:
Am 02.12.20 um 15:04 schrieb Seena Fallah:
I don't think so! I want to slow down the recovery not speed up and it says I should reduce these values.
I read the documentation the same. Low value = low weight, High value = high weight. [1]
Operations with higher weight get easier dispatched.
May I ask which exact issue are you seeing and which ceph version are you using?
Peter
[1] https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html/...
On 12/2/20 5:36 PM, Seena Fallah wrote:
If it uses PriorityQueue <https://en.wikipedia.org/wiki/Priority_queue#:~:text=In%20computer%20science%2C%20a%20priority,an%20element%20with%20low%20priority.> Data Structure an element with high priority should be dequeued before an element with low priority.
- osd client op priority The priority set for client operations. It is relative to osd recovery op priority. The op priority for client io is (default) 63. So with that regard a lower value should be lower priority. Same as the PriorityQueue you linked. A simple example would be a great way to take out any confusion. I'll try to attend a "Docubetter" meeting to discuss this. Gr. Stefan
FWIW https://github.com/ceph/ceph/blob/master/doc/dev/osd_internals/backfill_rese... has some discussion of op priorities, though client ops aren’t mentioned explicitly. If you like, enter a documentation tracker and tag me and I’ll look into adding that.
On Dec 2, 2020, at 9:56 AM, Stefan Kooman <stefan@bit.nl> wrote:
On 12/2/20 5:36 PM, Seena Fallah wrote:
If it uses PriorityQueue <https://en.wikipedia.org/wiki/Priority_queue#:~:text=In%20computer%20science%2C%20a%20priority,an%20element%20with%20low%20priority.> Data Structure an element with high priority should be dequeued before an element with low priority.
- osd client op priority The priority set for client operations. It is relative to osd recovery op priority.
The op priority for client io is (default) 63. So with that regard a lower value should be lower priority. Same as the PriorityQueue you linked.
A simple example would be a great way to take out any confusion. I'll try to attend a "Docubetter" meeting to discuss this.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sorry I got confused! Do you mean that both recovery_op_priority and recovery_priority should be 63 to have a slow recovery? If so why the client op priority is default 63 and recovery op is 3? This means that by default recovery op is more prioritize than client op! In the link you share (Github one) It says that if priority is higher as an integer it will mean that it prioritizes others! Can you please clarify this, please? I think this should be documented as well in Ceph because in general, I think it means that a higher integer number means higher priority than a lower integer number and so if we set recovery op priority to 63 it will do recovery too fast and will impact the overall cluster performance! Thanks. On Wed, Dec 2, 2020 at 10:25 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
FWIW https://github.com/ceph/ceph/blob/master/doc/dev/osd_internals/backfill_rese... has some discussion of op priorities, though client ops aren’t mentioned explicitly. If you like, enter a documentation tracker and tag me and I’ll look into adding that.
On Dec 2, 2020, at 9:56 AM, Stefan Kooman <stefan@bit.nl> wrote:
On 12/2/20 5:36 PM, Seena Fallah wrote:
If it uses PriorityQueue < https://en.wikipedia.org/wiki/Priority_queue#:~:text=In%20computer%20science%2C%20a%20priority,an%20element%20with%20low%20priority.> Data Structure an element with high priority should be dequeued before an element with low priority.
- osd client op priority The priority set for client operations. It is relative to osd recovery op priority.
The op priority for client io is (default) 63. So with that regard a lower value should be lower priority. Same as the PriorityQueue you linked.
A simple example would be a great way to take out any confusion. I'll try to attend a "Docubetter" meeting to discuss this.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sorry I got confused! Do you mean that both recovery_op_priority and recovery_priority should be 63 to have a slow recovery?
recovery_priority is set per-pool to rank recovery relative to other pools. This is not related to your questions. https://docs.ceph.com/en/latest/rados/operations/placement-groups/#prioritiz...
If so why the client op priority is default 63 and recovery op is 3? This means that by default recovery op is more prioritize than client op!
Exactly the opposite. Client ops take priority over recovery ops. And various other ops have priorities as described in the document I pointed you to.
In the link you share (Github one) It says that if priority is higher as an integer it will mean that it prioritizes others! Can you please clarify this, please? I think this should be documented as well in Ceph because in general, I think it means that a higher integer number means higher priority than a lower integer number and so if we set recovery op priority to 63 it will do recovery too fast and will impact the overall cluster performance!
That’s why it is commonly suggested to set recovery_op_priority to 1 if you need to slow down recovery as well as the other values I sent you. https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#operatio...
Thanks.
On Wed, Dec 2, 2020 at 10:25 PM Anthony D'Atri <anthony.datri@gmail.com> wrote: FWIW https://github.com/ceph/ceph/blob/master/doc/dev/osd_internals/backfill_rese... has some discussion of op priorities, though client ops aren’t mentioned explicitly. If you like, enter a documentation tracker and tag me and I’ll look into adding that.
On Dec 2, 2020, at 9:56 AM, Stefan Kooman <stefan@bit.nl> wrote:
On 12/2/20 5:36 PM, Seena Fallah wrote:
If it uses PriorityQueue <https://en.wikipedia.org/wiki/Priority_queue#:~:text=In%20computer%20science%2C%20a%20priority,an%20element%20with%20low%20priority.> Data Structure an element with high priority should be dequeued before an element with low priority.
- osd client op priority The priority set for client operations. It is relative to osd recovery op priority.
The op priority for client io is (default) 63. So with that regard a lower value should be lower priority. Same as the PriorityQueue you linked.
A simple example would be a great way to take out any confusion. I'll try to attend a "Docubetter" meeting to discuss this.
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If so why the client op priority is default 63 and recovery op is 3? This means that by default recovery op is more prioritize than client op!
Exactly the opposite. Client ops take priority over recovery ops. And various other ops have priorities as described in the document I pointed you to.
Now that I poke around, I do find some resources that claim that a *lower* osd_recovery_op_priority value *increases* the priority of recovery ops vs client ops, eg. this one: https://www.suse.com/support/kb/doc/?id=000019693 This is counter to everything I’ve always been told and observed. Poking around in the sources my should-be-in-bed eyes don’t find a definitive answer, but I do see in OpRequest.cc : if (req->get_priority() < tracker->cct->_conf->osd_client_op_priority) { // don't warn as quickly for low priority ops warn_interval_multiplier = tracker->cct->_conf->osd_recovery_op_warn_multiple; } which seems to corroborate the idea that a lower value is a lower priority. If we can get consensus here I’ll add a bit to the docs to make the relationship more clear.
Thanks. It seems it is related to wpq implementation on how it is organizing priorities! I want to slow down the keys/s and I've set all the priorities to 1 for recovery but it doesn't slow down! On Thu, Dec 3, 2020 at 1:13 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
If so why the client op priority is default 63 and recovery op is 3? This means that by default recovery op is more prioritize than client op!
Exactly the opposite. Client ops take priority over recovery ops. And various other ops have priorities as described in the document I pointed you to.
Now that I poke around, I do find some resources that claim that a *lower* osd_recovery_op_priority value *increases* the priority of recovery ops vs client ops, eg. this one:
https://www.suse.com/support/kb/doc/?id=000019693
This is counter to everything I’ve always been told and observed. Poking around in the sources my should-be-in-bed eyes don’t find a definitive answer, but I do see in OpRequest.cc :
if (req->get_priority() < tracker->cct->_conf->osd_client_op_priority) { // don't warn as quickly for low priority ops warn_interval_multiplier = tracker->cct->_conf->osd_recovery_op_warn_multiple; }
which seems to corroborate the idea that a lower value is a lower priority.
If we can get consensus here I’ll add a bit to the docs to make the relationship more clear.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
In certain cases (Luminous) it can actually be faster to destroy an OSD and recreate it than to let it backfill huge maps, but I think that’s been improved by Nautilus. You might also try setting osd_op_queue_cut_off = high to reduce the impact of recovery on client operations. This became the default setting at Octopus; yours is likely set to low.
I did the same but it moved 200K keys/s!
On Wed, Dec 2, 2020 at 5:14 PM Stefan Kooman <stefan@bit.nl> wrote:
On 12/1/20 12:37 AM, Seena Fallah wrote:
Hi all,
Is there any configuration to slow down keys/s in recovery mode?
Not just keys, but you can limit recovery / backfill like this:
ceph tell 'osd.*' injectargs '--osd_max_backfills 1' ceph tell 'osd.*' injectargs '--osd_recovery_max_active 1'
Gr. Stefan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Peter Lieven
-
Seena Fallah
-
Stefan Kooman