On 2020-08-18 11:13, Hans van den Bogert wrote: Not sure if I qualify for senior, but here are my 2 cents ... I would argue that you do want to do this in one step. Doing this in multiple steps will trigger data movement every time you change pg_num (and pgp_num for that matter). Ceph will recalculate a new mapping every time you change the pg(p)_num for a pool (or by altering CRUSH rules). osd_recovery_max_active = 1 osd_max_backfills = 1 If your cluster can't handle this than I wonder what a disk / host failure would trigger. Some on this list would argue that you also want the following setting to avoid client IO starvation: ceph config set osd osd_op_queue_cut_off high This is already the default in Octopus. Gr. Stefan

Hans, I made a big change in my staging cluster before, I set a pool pg_num from 8 to 2048, it cased the cluster available for a long time :( On 18/8/2020 下午5:13, Hans van den Bogert wrote: > I don't think it might lead to more client slow requests if you set it > to 4096 in one step, since there is a cap on how many > recovery/backfill requests there can be per OSD at any given time. > > I am not sure though, but I am happy to be proved wrong by the senior > members in this list :) > > Hans > > On 8/18/20 10:23 AM, norman wrote: >> Hi guys, >> >> I have a rbd pool pg_num 2048, I want to change it to 4096,  how can I >> do this? >> >> If I change it directly to 4096, it may cause the client slow requests, >> What the better step size should be? >> >> Thanks, >> >> Kern >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Only bad for the clients. ;-) As Stefan already advised, turn down these values to 1 and let the cluster rebalance slowly. If the client performance seems fine you can increase by 1 or so and see how it behaves. You'll have to find reasonable values for your specific setup to have a good mix between quick recovery without impacting client performance too much. Regards, Eugen Zitat von norman <norman.kern(a)gmx.com>om>: > Stefan, > > I agree with you about the crush rule,  but I truely met the problem for > the cluster, > > > > Kern > > On 18/8/2020 下午5:27, Stefan Kooman wrote: >> On 2020-08-18 11:13, Hans van den Bogert wrote: >>> I don't think it might lead to more client slow requests if you set it >>> to 4096 in one step, since there is a cap on how many recovery/backfill >>> requests there can be per OSD at any given time. >>> >>> I am not sure though, but I am happy to be proved wrong by the senior >>> members in this list :) >> Not sure if I qualify for senior, but here are my 2 cents ... >> >> I would argue that you do want to do this in one step. Doing this in >> multiple steps will trigger data movement every time you change pg_num >> (and pgp_num for that matter). Ceph will recalculate a new mapping every >> time you change the pg(p)_num for a pool (or by altering CRUSH rules). >> >> osd_recovery_max_active = 1 >> osd_max_backfills = 1 >> >> If your cluster can't handle this than I wonder what a disk / host >> failure would trigger. >> >> Some on this list would argue that you also want the following setting >> to avoid client IO starvation: >> >> ceph config set osd osd_op_queue_cut_off high >> >> This is already the default in Octopus. >> >> Gr. Stefan >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

To set these setting during runtime I use the following commands from my admin node: ceph tell osd.* injectargs '--osd-max-backfills 1' ceph tell osd.* injectargs '--osd-recovery-max-active 1' ceph tell osd.* injectargs '--osd-op-queue-cut-off high' Right? On Wed, Aug 19, 2020 at 11:08 AM Stefan Kooman <stefan(a)bit.nl> wrote: > > On 2020-08-19 04:05, norman wrote: > > Hans, > > > > I made a big change in my staging cluster before, I set a pool pg_num > > from 8 to 2048, it cased the cluster available for a long time :( > > I doubt it will have that big an effect this time. The change from 8 -> > 2048 is way bigger than "just" a doubling to 4096. It of course depends > on how much data has been added to the pool in the meantime. But if the > PGs hold less data per PG, the impact is probably lower. > > It's always a tradeoff between recovery speed and decent client IO > performance, and highly dependent per use case, cluster size, hardware > specifications like disk type, amount of RAM / CPU and networking. > > For some clusters the values you have used might by no big deal (if you > have hundreds of nodes) but for small(er) clusters this can have a big > impact. Might, if you have NVMe and loads of memory and CPU you might > even get away with it. > > So it's best to start with low(est) possible recovery/backfill settings > and slowly scaling up. Having metrics of your cluster and client VMs > (perceived latency on VMs for example) will be crucial to put a number > on this. Having a baseline of the performance of the cluster will help > to decide when things get out of hand. It will also help to identify the > time the cluster is less busy, and use that time window to perform > maintenance (i.e. not when backups are active, which tend to stress > clusters a lot). > > Gr. Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

On 2020-08-28 09:36, Martin Palma wrote: ceph tell 'osd.*' injectargs '--osd_recovery_max_active 1' ^^ That's what I found in our history. Not sure if the single quotes are needed around osd.*, but this certainly works. But I'm not sure if the settings can be applied without a restart as far as osd-op-queue-cut-off high is concerned. IIRC you can't change this live, and a OSD restart is needed. Gr. Stefan P.s. aOn any node that has admin keyring and cluster connectivity you can run these commands. Admin host seems like a right place to do so. Gr. Stefan

On 2020-08-28 12:57, Martin Palma wrote: Yes, I would do that. We have seen that after creating more PGs those PGs will get (deep-)scrubbed as they have never been (deep-)scrubbed before. Only enable (deep-)scubbing after all data has been moved and cluster is HEALTH_OK. Because it will take a long time before they are all (deep-)scrubbed. Make sure you have a "scrub window" that is convenient (low traffic). Quite a few PGs might get scrubbed on different OSDs at the same time. Reminds me for the need for a "scrub" module in the manager to handle this stuff properly ... as soon as I have some time I'll come up with a proposal. Gr. Stefan