No, the recipe I gave was for trying to recover healthy status of all PGs in the current situation. I would avoid moving OSDs at all cost, because it will always imply rebalancing. Any change to the crush map changes how PGs are hashed onto OSDs, which in turn triggers a rebalancing. If moving OSDs cannot be avoided, I usually do: - evacuate OSDs that need to move - move empty (!) OSDs to new location - let data move back onto OSDs There are other ways of doing it, with their own pro's and cons. For example, if your client load allows high-bandwidth rebuild operations, you can also - shut down OSDs that need to move (make sure you don't shut down too many from different failure domains at the same time) - let the remaining OSDs rebuild the missing data - after health is back to OK, move OSDs and start up The second way is usually faster, but has the drawback that new writes will go to less redundant storage for a while. The first method takes longer, but there is no redundancy degradation along the way. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Nico Schottelius <nico.schottelius@ungleich.ch> Sent: 22 September 2020 22:13:49 To: Frank Schilder Cc: Nico Schottelius; Andreas John; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Unknown PGs after osd move Hey Frank, Frank Schilder <frans@dtu.dk> writes:
Is the crush map aware about that?
Yes, it correctly shows the osds at serve8 (previously server15).
I didn't ever try that, but don't you need to cursh move it?
I originally imagined this, too. But as soon as the osd starts on a new server it is automatically put into the serve8 bucket.
It does not work like this, unfortunately. If you physically move disks to a new server without "informing ceph" in advance, hat is, crush move the OSD while they are up, ceph looses placement information. You can post-repair such a situation by temporarily "crush moving" (software move, not hardware move) the OSDs back to their previous host buckets, wait for peering to complete, and then "crush move" them to their new location again.
That is good to know. So in theory: - crush move osd to a different server bucket - shutdown osd - move physically to another server - no rebalancing needed Should do the job? It won't accept today's rebalance, but it would be good to have a sane way for the future. Cheers, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch