A few years ago Dan van der Ster and I were working on two similar scripts for increasing pgs. Just have a look at the following link: https://github.com/cernceph/ceph-scripts/blob/master/tools/split/ceph-gentle... ___________________________________ Clyso GmbH Am 18.08.2020 um 11:27 schrieb Stefan Kooman:

On 2020-08-18 11:13, Hans van den Bogert wrote:

...

I don't think it might lead to more client slow requests if you set it to 4096 in one step, since there is a cap on how many recovery/backfill requests there can be per OSD at any given time.

I am not sure though, but I am happy to be proved wrong by the senior members in this list :) Not sure if I qualify for senior, but here are my 2 cents ...

I would argue that you do want to do this in one step. Doing this in multiple steps will trigger data movement every time you change pg_num (and pgp_num for that matter). Ceph will recalculate a new mapping every time you change the pg(p)_num for a pool (or by altering CRUSH rules).

osd_recovery_max_active = 1 osd_max_backfills = 1

If your cluster can't handle this than I wonder what a disk / host failure would trigger.

Some on this list would argue that you also want the following setting to avoid client IO starvation:

ceph config set osd osd_op_queue_cut_off high

This is already the default in Octopus.

Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I set the values large for a quick recover:

osd_recovery_max_active 16

osd_max_backfills 32

Is it a very bad setting?

Only bad for the clients. ;-) As Stefan already advised, turn down these values to 1 and let the cluster rebalance slowly. If the client performance seems fine you can increase by 1 or so and see how it behaves. You'll have to find reasonable values for your specific setup to have a good mix between quick recovery without impacting client performance too much. Regards, Eugen Zitat von norman <norman.kern@gmx.com>:

Stefan,

I agree with you about the crush rule,  but I truely met the problem for the cluster,

I set the values large for a quick recover:

osd_recovery_max_active 16

osd_max_backfills 32

Is it a very bad setting?

Kern

On 18/8/2020 下午5:27, Stefan Kooman wrote:

...

On 2020-08-18 11:13, Hans van den Bogert wrote:

...

I don't think it might lead to more client slow requests if you set it to 4096 in one step, since there is a cap on how many recovery/backfill requests there can be per OSD at any given time.

I am not sure though, but I am happy to be proved wrong by the senior members in this list :) Not sure if I qualify for senior, but here are my 2 cents ...

I would argue that you do want to do this in one step. Doing this in multiple steps will trigger data movement every time you change pg_num (and pgp_num for that matter). Ceph will recalculate a new mapping every time you change the pg(p)_num for a pool (or by altering CRUSH rules).

osd_recovery_max_active = 1 osd_max_backfills = 1

If your cluster can't handle this than I wonder what a disk / host failure would trigger.

Some on this list would argue that you also want the following setting to avoid client IO starvation:

ceph config set osd osd_op_queue_cut_off high

This is already the default in Octopus.

Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

On 2020-08-19 04:05, norman wrote:

Hans,

I made a big change in my staging cluster before, I set a pool pg_num from 8 to 2048, it cased the cluster available for a long time :(

I doubt it will have that big an effect this time. The change from 8 -> 2048 is way bigger than "just" a doubling to 4096. It of course depends on how much data has been added to the pool in the meantime. But if the PGs hold less data per PG, the impact is probably lower. It's always a tradeoff between recovery speed and decent client IO performance, and highly dependent per use case, cluster size, hardware specifications like disk type, amount of RAM / CPU and networking. For some clusters the values you have used might by no big deal (if you have hundreds of nodes) but for small(er) clusters this can have a big impact. Might, if you have NVMe and loads of memory and CPU you might even get away with it. So it's best to start with low(est) possible recovery/backfill settings and slowly scaling up. Having metrics of your cluster and client VMs (perceived latency on VMs for example) will be crucial to put a number on this. Having a baseline of the performance of the cluster will help to decide when things get out of hand. It will also help to identify the time the cluster is less busy, and use that time window to perform maintenance (i.e. not when backups are active, which tend to stress clusters a lot). Gr. Stefan

If I remember correctly op-queue-cut-off can't be changed during runtime, the other two can. Zitat von Martin Palma <martin@palma.bz>:

To set these setting during runtime I use the following commands from my admin node:

ceph tell osd.* injectargs '--osd-max-backfills 1' ceph tell osd.* injectargs '--osd-recovery-max-active 1' ceph tell osd.* injectargs '--osd-op-queue-cut-off high'

Right?

On Wed, Aug 19, 2020 at 11:08 AM Stefan Kooman <stefan@bit.nl> wrote:

...

On 2020-08-19 04:05, norman wrote:

...

Hans,

I made a big change in my staging cluster before, I set a pool pg_num from 8 to 2048, it cased the cluster available for a long time :(

I doubt it will have that big an effect this time. The change from 8 -> 2048 is way bigger than "just" a doubling to 4096. It of course depends on how much data has been added to the pool in the meantime. But if the PGs hold less data per PG, the impact is probably lower.

It's always a tradeoff between recovery speed and decent client IO performance, and highly dependent per use case, cluster size, hardware specifications like disk type, amount of RAM / CPU and networking.

For some clusters the values you have used might by no big deal (if you have hundreds of nodes) but for small(er) clusters this can have a big impact. Might, if you have NVMe and loads of memory and CPU you might even get away with it.

So it's best to start with low(est) possible recovery/backfill settings and slowly scaling up. Having metrics of your cluster and client VMs (perceived latency on VMs for example) will be crucial to put a number on this. Having a baseline of the performance of the cluster will help to decide when things get out of hand. It will also help to identify the time the cluster is less busy, and use that time window to perform maintenance (i.e. not when backups are active, which tend to stress clusters a lot).

Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Thanks for the clarification. And by raising the PG should we also set "no scrub" and "no deep scrub" during data movement? What is here the recommandation? On Fri, Aug 28, 2020 at 10:25 AM Stefan Kooman <stefan@bit.nl> wrote:

On 2020-08-28 09:36, Martin Palma wrote:

...

To set these setting during runtime I use the following commands from my admin node:

ceph tell osd.* injectargs '--osd-max-backfills 1' ceph tell osd.* injectargs '--osd-recovery-max-active 1' ceph tell osd.* injectargs '--osd-op-queue-cut-off high'

ceph tell 'osd.*' injectargs '--osd_recovery_max_active 1'

^^ That's what I found in our history. Not sure if the single quotes are needed around osd.*, but this certainly works.

But I'm not sure if the settings can be applied without a restart as far as osd-op-queue-cut-off high is concerned. IIRC you can't change this live, and a OSD restart is needed.

Gr. Stefan

P.s. aOn any node that has admin keyring and cluster connectivity you can run these commands. Admin host seems like a right place to do so.

Gr. Stefan