Hi Eugen, I've removed 12 OSDs with a 'ceph orch osd rm ID --replace' last week on Pacific and even though only 10 OSDs started draining their PGs at a time (the other 2 waiting for an available 'slot', obviously) all 12 OSDs got removed successfully at the end. Cheers, Frédéric. ----- Le 16 Nov 24, à 15:08, Eugen Block eblock@nde.ag a écrit :
I have a feeling that this could be related to the drives, but I have no real proof. I drained the SSD OSDs yesterday, hours later I wanted to remove the OSDs (no PGs were on them anymore) via for loop with the orchestrator (ceph orch osd rm ID --force --zap). The first one got removed quite quickly, but the others disappeared from the queue. I tried again and looked at the iostat output on the node where the queued OSD was running on. The drive had no IO at all but was utilized 100% for several minutes until it was eventually removed. I find that very weird, especially since we’re currently helping a customer rebuild OSDs on Pacific as well, and I haven’t seen such a behavior yet. And we already redeployed 132 OSDs so far.
Zitat von Eugen Block <eblock@nde.ag>:
Hi,
I'm not sure if this has been asked before, or if there's an existing tracker issue already. It's difficult to reproduce it on my lab clusters. I'm testing some new SSD OSDs on a Pacific cluster (16.2.15) and noticed that if we instruct the orchestrator to remove two or three OSDs (issuing the command 'ceph orch osd rm {ID}' a couple of times), it eventually only removes the first in the queue. I've been watching 'ceph orch osd rm status' to see the progress, and then the rest of the queued OSDs suddenly vanish from the status and never get removed. Then I have to issue the command again. If I remove one OSD by one, so no others in the queue, they are all successfully removed. Why is this happening? Is this a bug in Pacific? Has someone seen this in newer releases?
Thanks! Eugen
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io