Hi Jesper, could you please provide more details about the cluster (the usual like 'ceph osd tree', 'ceph osd df', 'ceph versions')? I find it unusual to enable maintenance mode to add OSDs, is there a specific reason? And why adding OSDs manually with 'ceph orch osd add', why not have a spec file applicable to the host and let it deploy them automatically?
1. Can we bulk change weights on all existing OSD’s without huge data movement? If so – how?
This could do the trick, but not without data movement. ceph osd crush reweight-subtree <name> <weight>
2. Can we ceph orch add – but with a specific initial weight?
You can set an initial crush weight, for example we set it to 0 years ago to control when the backfill will start: ceph config get osd osd_crush_initial_weight 0.000000 But the questions is why you have to deal with weights in such a manner, can you provide mode details what exactly goes wrong if you add OSDs with their target weight? Thanks, Eugen Zitat von "Jesper Agerbo Krogh [JSKR]" <JSKR@topsoe.com>:
Hi All
We have a nicely functional NVMe cluster running – but in the process of expanding it we have encountered slow_ops
The drill was:
1. Maintaince mode 2. Ceph orch osd add 3. Change weights to 1 4. Disable maintaince mode.
Somethings goes wrong around 3 – where slowops begins to kick in – but restarting the OSD’s underneath can make it go away – tested twice. In one situation the 2) step failed to bring up the OSD – causing the entire process to work 100% correctly. Thus changing weight from 7.68 to 1 In above process on live OSD seem to be the issue.
It is actually a flaw that we managed to get the entire cluster created with weight 1 instead of 7.68 in the first place thus:
1. Can we bulk change weights on all existing OSD’s without huge data movement? If so – how? 2. Can we ceph orch add – but with a specific initial weight?
Thanks.
Best regards, Jesper Agerbo Krogh Director Digitalization Digitalization
[cid:image001.png@01DA794D.5131C5B0]<http://www.linkedin.com/company/haldor-topsoe> Topsoe A/S Haldor Topsøes Allé 1 2800 Kgs. Lyngby Denmark Phone (direct): 27773240 [cid:image002.jpg@01DA794D.5131C5B0]<http://www.linkedin.com/company/haldor-topsoe> [cid:image003.jpg@01DA794D.5131C5B0] <https://twitter.com/topsoe_official> [cid:image004.jpg@01DA794D.5131C5B0] <https://www.facebook.com/TopsoeOfficial> [cid:image005.jpg@01DA794D.5131C5B0] <https://www.topsoe.com/our-resources/knowledge/videos/> Read more at topsoe.com<http://www.topsoe.com>
Topsoe A/S and/or its affiliates. This e-mail message (including attachments, if any) is confidential and may be privileged. It is intended only for the addressee. Any unauthorised distribution or disclosure is prohibited. Disclosure to anyone other than the intended recipient does not constitute waiver of privilege. If you have received this email in error, please notify the sender by email and delete it and any attachments from your computer system and records.