1 Dec
2020
1 Dec
'20
12:37 a.m.
Hi all,
Is there any configuration to slow down keys/s in recovery mode?
Thanks.
2 Dec
2 Dec
2:44 p.m.
On 12/1/20 12:37 AM, Seena Fallah wrote:
Hi all,
Is there any configuration to slow down keys/s in recovery mode?
Not just keys, but you can limit recovery / backfill like this:
ceph tell 'osd.*' injectargs '--osd_max_backfills 1'
ceph tell 'osd.*' injectargs '--osd_recovery_max_active 1'
Gr. Stefan
2:46 p.m.
I did the same but it moved 200K keys/s!
On Wed, Dec 2, 2020 at 5:14 PM Stefan Kooman <stefan(a)bit.nl> wrote:
On 12/1/20 12:37 AM, Seena Fallah wrote:
Hi all,
Is there any configuration to slow down keys/s in recovery mode?
Not just keys, but you can limit recovery / backfill like this:
ceph tell 'osd.*' injectargs '--osd_max_backfills 1'
ceph tell 'osd.*' injectargs '--osd_recovery_max_active 1'
Gr. Stefan
2:55 p.m.
This is what I used in recovery:
osd max backfills = 1
osd recovery max active = 1
osd recovery op priority = 1
osd recovery priority = 1
osd recovery sleep ssd = 0.2
But it doesn't help much!
On Wed, Dec 2, 2020 at 5:23 PM Stefan Kooman <stefan(a)bit.nl> wrote:
On 12/2/20 2:46 PM, Seena Fallah wrote:
I did the same but it moved 200K keys/s!
You might also want to decrease the op priority (as in _increasing_ the
number) of "osd_recovery_op_priority".
Gr. Stefan
3:01 p.m.
On 12/2/20 2:55 PM, Seena Fallah wrote:
This is what I used in recovery:
osd max backfills = 1
osd recovery max active = 1
osd recovery op priority = 1
^^ Shouldn't this go to 63 instead of 1? At least if I read this post
from SUSE correctly I think it should [1].
osd recovery priority = 1
^^ same for this.
osd recovery sleep ssd = 0.2
Gr. Stefan
[1]: https://www.suse.com/support/kb/doc/?id=000019693
3:04 p.m.
I don't think so! I want to slow down the recovery not speed up and it says
I should reduce these values.
On Wed, Dec 2, 2020 at 5:31 PM Stefan Kooman <stefan(a)bit.nl> wrote:
On 12/2/20 2:55 PM, Seena Fallah wrote:
This is what I used in recovery:
osd max backfills = 1
osd recovery max active = 1
osd recovery op priority = 1
^^ Shouldn't this go to 63 instead of 1? At least if I read this post
from SUSE correctly I think it should [1].
osd recovery priority = 1
^^ same for this.
osd recovery sleep ssd = 0.2
Gr. Stefan
[1]: https://www.suse.com/support/kb/doc/?id=000019693
3:22 p.m.
On 12/2/20 3:04 PM, Seena Fallah wrote:
I don't think so! I want to slow down the recovery
not speed up and it
says I should reduce these values.
osd recovery op priority: This is the priority set for recovery
operation. Lower the number, higher the recovery priority. Higher
recovery priority might cause performance degradation until recovery
completes.
"Lower the number, higher the recovery priority." With that reasoning
"higher the number, lower the recovery priority" should also be true.
I would try "63" for both priorities.
Gr. Stefan
4:34 p.m.
Am 02.12.20 um 15:04 schrieb Seena Fallah:
I don't think so! I want to slow down the recovery
not speed up and it says
I should reduce these values.
I read the documentation the same. Low value = low weight, High value = high weight. [1]
Operations with higher weight get easier dispatched.
May I ask which exact issue are you seeing and which ceph version are you using?
Peter
[1]
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html…
5:02 p.m.
"Higher recovery priority might cause performance degradation until
recovery completes."
But what about this statement? I found this that it means if I set priority
to 63, I will lose the cluster performance for clients. Am I wrong? Does it
mean the performance for recovery?
I'm using nautilus 14.2.14.
[1]
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html…
Can you please point me to the correct section. I don't find the point of
this doc.
On Wed, Dec 2, 2020 at 7:04 PM Peter Lieven <pl(a)kamp.de> wrote:
> Am 02.12.20 um 15:04 schrieb Seena Fallah:
> > I don't think so! I want to slow down the recovery not speed up and it
> says
> > I should reduce these values.
>
>
> I read the documentation the same. Low value = low weight, High value =
> high weight. [1]
>
> Operations with higher weight get easier dispatched.
>
>
> May I ask which exact issue are you seeing and which ceph version are you
> using?
>
>
> Peter
>
>
[1]
>
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html…
>
>
>
>
5:36 p.m.
If it uses PriorityQueue
<https://en.wikipedia.org/wiki/Priority_queue#:~:text=In%20computer%20science%2C%20a%20priority,an%20element%20with%20low%20priority.>
Data
Structure an element with high priority should be dequeued before an
element with low priority.
On Wed, Dec 2, 2020 at 7:32 PM Seena Fallah <seenafallah(a)gmail.com> wrote:
"Higher recovery priority might cause performance
degradation until
recovery completes."
But what about this statement? I found this that it means if I set
priority to 63, I will lose the cluster performance for clients. Am I
wrong? Does it mean the performance for recovery?
I'm using nautilus 14.2.14.
[1]
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html…
Can you please point me to the correct section. I don't find the point of
this doc.
On Wed, Dec 2, 2020 at 7:04 PM Peter Lieven <pl(a)kamp.de> wrote:
> Am 02.12.20 um 15:04 schrieb Seena Fallah:
> > I don't think so! I want to slow down the recovery not speed up and it
> says
> > I should reduce these values.
>
>
> I read the documentation the same. Low value = low weight, High value =
> high weight. [1]
>
> Operations with higher weight get easier dispatched.
>
>
> May I ask which exact issue are you seeing and which ceph version are you
> using?
>
>
> Peter
>
>
> [1]
>
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.3/html…
>
>
>
>
6:56 p.m.
On 12/2/20 5:36 PM, Seena Fallah wrote:
If it uses PriorityQueue
<https://en.wikipedia.org/wiki/Priority_queue#:~:text=In%20computer%20science%2C%20a%20priority,an%20element%20with%20low%20priority.> Data
Structure an element with high priority should be dequeued before an
element with low priority.
- osd client op priority
The priority set for client operations. It is relative to osd recovery
op priority.
The op priority for client io is (default) 63. So with that regard a
lower value should be lower priority. Same as the PriorityQueue you linked.
A simple example would be a great way to take out any confusion. I'll
try to attend a "Docubetter" meeting to discuss this.
Gr. Stefan
7:55 p.m.
FWIW
https://github.com/ceph/ceph/blob/master/doc/dev/osd_internals/backfill_res…
has some discussion of op priorities, though client ops aren’t mentioned explicitly. If
you like, enter a documentation tracker and tag me and I’ll look into adding that.
On Dec 2, 2020, at 9:56 AM, Stefan Kooman
<stefan(a)bit.nl> wrote:
On 12/2/20 5:36 PM, Seena Fallah wrote:
If it uses PriorityQueue
<https://en.wikipedia.org/wiki/Priority_queue#:~:text=In%20computer%20science%2C%20a%20priority,an%20element%20with%20low%20priority.>
Data Structure an element with high priority should be dequeued before an element with low
priority.
- osd client op priority
The priority set for client operations. It is relative to osd recovery op priority.
The op priority for client io is (default) 63. So with that regard a lower value should
be lower priority. Same as the PriorityQueue you linked.
A simple example would be a great way to take out any confusion. I'll try to attend a
"Docubetter" meeting to discuss this.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
9:05 p.m.
Sorry I got confused! Do you mean that both recovery_op_priority and
recovery_priority should be 63 to have a slow recovery?
If so why the client op priority is default 63 and recovery op is 3? This
means that by default recovery op is more prioritize than client op!
In the link you share (Github one) It says that if priority is higher as an
integer it will mean that it prioritizes others!
Can you please clarify this, please? I think this should be documented as
well in Ceph because in general, I think it means that a higher integer
number means higher priority than a lower integer number and so if we set
recovery op priority to 63 it will do recovery too fast and will impact the
overall cluster performance!
Thanks.
On Wed, Dec 2, 2020 at 10:25 PM Anthony D'Atri <anthony.datri(a)gmail.com>
wrote:
FWIW
https://github.com/ceph/ceph/blob/master/doc/dev/osd_internals/backfill_res…
has some discussion of op priorities, though client ops aren’t mentioned
explicitly. If you like, enter a documentation tracker and tag me and I’ll
look into adding that.
On Dec 2, 2020, at 9:56 AM, Stefan Kooman
<stefan(a)bit.nl> wrote:
On 12/2/20 5:36 PM, Seena Fallah wrote:
> If it uses PriorityQueue <
https://en.wikipedia.org/wiki/Priority_queue#:~:text=In%20computer%20scienc…
Data Structure an element with high priority should be dequeued before an
element with low priority.
- osd client op priority
The priority set for client operations. It is relative to osd recovery
op
priority.
The op priority for client io is (default) 63. So with that regard a
lower value
should be lower priority. Same as the PriorityQueue you linked.
A simple example would be a great way to take out any confusion. I'll
try to
attend a "Docubetter" meeting to discuss this.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3 Dec
3 Dec
9:46 a.m.
Sorry I got confused! Do you mean that both recovery_op_priority and recovery_priority
should be 63 to have a slow recovery?
recovery_priority is set per-pool to rank recovery relative to other pools. This is not
related to your questions.
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#prioriti…
If so why the client op priority is default 63 and
recovery op is 3? This means that by default recovery op is more prioritize than client
op!
Exactly the opposite. Client ops take priority over recovery ops. And various other ops
have priorities as described in the document I pointed you to.
In the link you share (Github one) It says that if
priority is higher as an integer it will mean that it prioritizes others!
Can you please clarify this, please? I think this should be documented as well in Ceph
because in general, I think it means that a higher integer number means higher priority
than a lower integer number and so if we set recovery op priority to 63 it will do
recovery too fast and will impact the overall cluster performance!
That’s why it is commonly suggested to set recovery_op_priority to 1 if you need to slow
down recovery as well as the other values I sent you.
https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#operati…
Thanks.
On Wed, Dec 2, 2020 at 10:25 PM Anthony D'Atri <anthony.datri(a)gmail.com>
wrote:
FWIW
https://github.com/ceph/ceph/blob/master/doc/dev/osd_internals/backfill_res…
has some discussion of op priorities, though client ops aren’t mentioned explicitly. If
you like, enter a documentation tracker and tag me and I’ll look into adding that.
On Dec 2, 2020, at 9:56 AM, Stefan Kooman
<stefan(a)bit.nl> wrote:
On 12/2/20 5:36 PM, Seena Fallah wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io If it uses PriorityQueue
<https://en.wikipedia.org/wiki/Priority_queue#:~:text=In%20computer%20science%2C%20a%20priority,an%20element%20with%20low%20priority.>
Data Structure an element with high priority should be dequeued before an element with low
priority.
- osd client op priority
The priority set for client operations. It is relative to osd recovery op priority.
The op priority for client io is (default) 63. So with that regard a lower value should
be lower priority. Same as the PriorityQueue you linked.
A simple example would be a great way to take out any confusion. I'll try to attend a
"Docubetter" meeting to discuss this.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:41 a.m.
If so why the
client op priority is default 63 and recovery op is 3? This means that by default recovery
op is more prioritize than client op!
Exactly the opposite. Client ops take priority over recovery ops. And various other ops
have priorities as described in the document I pointed you to.
3:41 p.m.
Thanks. It seems it is related to wpq implementation on how it is
organizing priorities!
I want to slow down the keys/s and I've set all the priorities to 1 for
recovery but it doesn't slow down!
On Thu, Dec 3, 2020 at 1:13 PM Anthony D'Atri <anthony.datri(a)gmail.com>
wrote:
> If so why the client op priority is default
63 and recovery op is 3?
This means that by default recovery op is more prioritize
than client op!
Exactly the opposite. Client ops take priority over recovery ops. And
various
other ops have priorities as described in the document I pointed
you to.
Now that I poke around, I do find some resources that claim that a *lower*
osd_recovery_op_priority value *increases* the priority of recovery ops vs
client ops, eg. this one:
https://www.suse.com/support/kb/doc/?id=000019693
This is counter to everything I’ve always been told and observed. Poking
around in the sources my should-be-in-bed eyes don’t find a definitive
answer, but I do see in OpRequest.cc :
if (req->get_priority() < tracker->cct->_conf->osd_client_op_priority)
{
// don't warn as quickly for low priority ops
warn_interval_multiplier =
tracker->cct->_conf->osd_recovery_op_warn_multiple;
}
which seems to corroborate the idea that a lower value is a lower priority.
If we can get consensus here I’ll add a bit to the docs to make the
relationship more clear.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2 Dec
2 Dec
5:39 p.m.
In certain cases (Luminous) it can actually be faster to destroy an OSD and recreate it
than to let it backfill huge maps, but I think that’s been improved by Nautilus.
You might also try setting
osd_op_queue_cut_off = high
to reduce the impact of recovery on client operations. This became the default setting at
Octopus; yours is likely set to low.
I did the same but it moved 200K keys/s!
On Wed, Dec 2, 2020 at 5:14 PM Stefan Kooman <stefan(a)bit.nl> wrote:
On 12/1/20 12:37 AM, Seena Fallah wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io Hi all,
Is there any configuration to slow down keys/s in recovery mode?
Not just keys, but you can limit recovery / backfill like this:
ceph tell 'osd.*' injectargs '--osd_max_backfills 1'
ceph tell 'osd.*' injectargs '--osd_recovery_max_active 1'
Gr. Stefan
1233
days inactive
1236
days old
17 comments
4 participants
participants (4)
-
Anthony D'Atri -
Peter Lieven -
Seena Fallah -
Stefan Kooman