Hi,
Thank you Eugen for pointing it out. Yes, OSDs are deployed by cephadm with drive_group. It seems that the orch module simplifies the process to make it easier for users.
yes it does, you can also manage your OSDs via dashboard which makes it also easier for some users.
When replacing an osd, there will be no PG remapping, and backfill will restore the data on the new disk, right?
That depends on how you decide to go through the replacement process. Usually without your intervention (e.g. setting the appropriate OSD flags) the remapping will happen after an OSD goes down and out.
In case of restoring a host with multiple OSDs, like WAL/DB SSD needs to be replaced, I see two options. 1) Keep cluster in degraded state and rebuild all OSDs. 2) Mark those OSDs out so PGs are rebalanced, rebuild OSDs, and bring them back in to rebalance PGs again.
These are basically your options, yes.
The key here is how much time backfilling and rebalancing will take? The intention is to not keep cluster in degraded state for too long. I assume they are similar, because either of them is to copy the same amount of data? If that's true, then option #2 is pointless. Could anyone share such experiences, like how long time it takes to recover how much data on what kind of networking/computing env?
No, option 2 is not pointless, it helps you prevent a degraded state. Having a small cluster or crush rules that only allow few failed OSDs it could be dangerous taking out an entire node, risking another failure and potential data loss. It highly depends on your specific setup and if you're willing to take the risk during rebuild of a node. The recovery/backfill speed is also depeneding on the size of OSDs, the object sizes, amount of data, etc. You would probably need to search the mailing list for examples from someone sharing their experience or so, I don't have captured such statistics. Regards, Eugen Zitat von Tony Liu <tonyliu0592@hotmail.com>:
Thank you Eugen for pointing it out. Yes, OSDs are deployed by cephadm with drive_group. It seems that the orch module simplifies the process to make it easier for users.
When replacing an osd, there will be no PG remapping, and backfill will restore the data on the new disk, right?
In case of restoring a host with multiple OSDs, like WAL/DB SSD needs to be replaced, I see two options. 1) Keep cluster in degraded state and rebuild all OSDs. 2) Mark those OSDs out so PGs are rebalanced, rebuild OSDs, and bring them back in to rebalance PGs again. The key here is how much time backfilling and rebalancing will take? The intention is to not keep cluster in degraded state for too long. I assume they are similar, because either of them is to copy the same amount of data? If that's true, then option #2 is pointless. Could anyone share such experiences, like how long time it takes to recover how much data on what kind of networking/computing env?
Thanks! Tony
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, November 25, 2020 1:49 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: replace osd with Octopus
Hi,
assuming you deployed with cephadm since you're mentioning Octopus there's a brief section in [1]. The basis for the OSD deployment is the drive_group configuration. If nothing has changed in your setup and you replace an OSD cephadm will detect the available disk and match it with the drive_group config. If there's enough space on the SSD too, it will redeploy the OSD.
The same goes for your second case: you'll need to remove all OSDs from that host, zap the devices, replace the SSD and then cephadm will deploy the entire host. That's the simple case. If redeploying all OSDs on that host is not an option you'll probably have to pause orchestrator in order to migrate devices yourself to prevent to much data movement.
Regards, Eugen
[1] https://docs.ceph.com/en/latest/mgr/orchestrator/#replace-an-osd
Zitat von Tony Liu <tonyliu0592@hotmail.com>:
Hi,
I did some search about replacing osd, and found some different steps, probably for different release? Is there recommended process to replace an osd with Octopus? Two cases here: 1) replace HDD whose WAL and DB are on a SSD. 1-1) failed disk is replaced by the same model. 1-2) working disk is replaced by bigger one. 2) replace the SSD holding WAL and DB for multiple HDDs.
Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io