replace osd with Octopus
Hi, I did some search about replacing osd, and found some different steps, probably for different release? Is there recommended process to replace an osd with Octopus? Two cases here: 1) replace HDD whose WAL and DB are on a SSD. 1-1) failed disk is replaced by the same model. 1-2) working disk is replaced by bigger one. 2) replace the SSD holding WAL and DB for multiple HDDs. Thanks! Tony
Hi, assuming you deployed with cephadm since you're mentioning Octopus there's a brief section in [1]. The basis for the OSD deployment is the drive_group configuration. If nothing has changed in your setup and you replace an OSD cephadm will detect the available disk and match it with the drive_group config. If there's enough space on the SSD too, it will redeploy the OSD. The same goes for your second case: you'll need to remove all OSDs from that host, zap the devices, replace the SSD and then cephadm will deploy the entire host. That's the simple case. If redeploying all OSDs on that host is not an option you'll probably have to pause orchestrator in order to migrate devices yourself to prevent to much data movement. Regards, Eugen [1] https://docs.ceph.com/en/latest/mgr/orchestrator/#replace-an-osd Zitat von Tony Liu <tonyliu0592@hotmail.com>:
Hi,
I did some search about replacing osd, and found some different steps, probably for different release? Is there recommended process to replace an osd with Octopus? Two cases here: 1) replace HDD whose WAL and DB are on a SSD. 1-1) failed disk is replaced by the same model. 1-2) working disk is replaced by bigger one. 2) replace the SSD holding WAL and DB for multiple HDDs.
Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you Eugen for pointing it out. Yes, OSDs are deployed by cephadm with drive_group. It seems that the orch module simplifies the process to make it easier for users. When replacing an osd, there will be no PG remapping, and backfill will restore the data on the new disk, right? In case of restoring a host with multiple OSDs, like WAL/DB SSD needs to be replaced, I see two options. 1) Keep cluster in degraded state and rebuild all OSDs. 2) Mark those OSDs out so PGs are rebalanced, rebuild OSDs, and bring them back in to rebalance PGs again. The key here is how much time backfilling and rebalancing will take? The intention is to not keep cluster in degraded state for too long. I assume they are similar, because either of them is to copy the same amount of data? If that's true, then option #2 is pointless. Could anyone share such experiences, like how long time it takes to recover how much data on what kind of networking/computing env? Thanks! Tony
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, November 25, 2020 1:49 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: replace osd with Octopus
Hi,
assuming you deployed with cephadm since you're mentioning Octopus there's a brief section in [1]. The basis for the OSD deployment is the drive_group configuration. If nothing has changed in your setup and you replace an OSD cephadm will detect the available disk and match it with the drive_group config. If there's enough space on the SSD too, it will redeploy the OSD.
The same goes for your second case: you'll need to remove all OSDs from that host, zap the devices, replace the SSD and then cephadm will deploy the entire host. That's the simple case. If redeploying all OSDs on that host is not an option you'll probably have to pause orchestrator in order to migrate devices yourself to prevent to much data movement.
Regards, Eugen
[1] https://docs.ceph.com/en/latest/mgr/orchestrator/#replace-an-osd
Zitat von Tony Liu <tonyliu0592@hotmail.com>:
Hi,
I did some search about replacing osd, and found some different steps, probably for different release? Is there recommended process to replace an osd with Octopus? Two cases here: 1) replace HDD whose WAL and DB are on a SSD. 1-1) failed disk is replaced by the same model. 1-2) working disk is replaced by bigger one. 2) replace the SSD holding WAL and DB for multiple HDDs.
Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
Thank you Eugen for pointing it out. Yes, OSDs are deployed by cephadm with drive_group. It seems that the orch module simplifies the process to make it easier for users.
yes it does, you can also manage your OSDs via dashboard which makes it also easier for some users.
When replacing an osd, there will be no PG remapping, and backfill will restore the data on the new disk, right?
That depends on how you decide to go through the replacement process. Usually without your intervention (e.g. setting the appropriate OSD flags) the remapping will happen after an OSD goes down and out.
In case of restoring a host with multiple OSDs, like WAL/DB SSD needs to be replaced, I see two options. 1) Keep cluster in degraded state and rebuild all OSDs. 2) Mark those OSDs out so PGs are rebalanced, rebuild OSDs, and bring them back in to rebalance PGs again.
These are basically your options, yes.
The key here is how much time backfilling and rebalancing will take? The intention is to not keep cluster in degraded state for too long. I assume they are similar, because either of them is to copy the same amount of data? If that's true, then option #2 is pointless. Could anyone share such experiences, like how long time it takes to recover how much data on what kind of networking/computing env?
No, option 2 is not pointless, it helps you prevent a degraded state. Having a small cluster or crush rules that only allow few failed OSDs it could be dangerous taking out an entire node, risking another failure and potential data loss. It highly depends on your specific setup and if you're willing to take the risk during rebuild of a node. The recovery/backfill speed is also depeneding on the size of OSDs, the object sizes, amount of data, etc. You would probably need to search the mailing list for examples from someone sharing their experience or so, I don't have captured such statistics. Regards, Eugen Zitat von Tony Liu <tonyliu0592@hotmail.com>:
Thank you Eugen for pointing it out. Yes, OSDs are deployed by cephadm with drive_group. It seems that the orch module simplifies the process to make it easier for users.
When replacing an osd, there will be no PG remapping, and backfill will restore the data on the new disk, right?
In case of restoring a host with multiple OSDs, like WAL/DB SSD needs to be replaced, I see two options. 1) Keep cluster in degraded state and rebuild all OSDs. 2) Mark those OSDs out so PGs are rebalanced, rebuild OSDs, and bring them back in to rebalance PGs again. The key here is how much time backfilling and rebalancing will take? The intention is to not keep cluster in degraded state for too long. I assume they are similar, because either of them is to copy the same amount of data? If that's true, then option #2 is pointless. Could anyone share such experiences, like how long time it takes to recover how much data on what kind of networking/computing env?
Thanks! Tony
-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, November 25, 2020 1:49 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: replace osd with Octopus
Hi,
assuming you deployed with cephadm since you're mentioning Octopus there's a brief section in [1]. The basis for the OSD deployment is the drive_group configuration. If nothing has changed in your setup and you replace an OSD cephadm will detect the available disk and match it with the drive_group config. If there's enough space on the SSD too, it will redeploy the OSD.
The same goes for your second case: you'll need to remove all OSDs from that host, zap the devices, replace the SSD and then cephadm will deploy the entire host. That's the simple case. If redeploying all OSDs on that host is not an option you'll probably have to pause orchestrator in order to migrate devices yourself to prevent to much data movement.
Regards, Eugen
[1] https://docs.ceph.com/en/latest/mgr/orchestrator/#replace-an-osd
Zitat von Tony Liu <tonyliu0592@hotmail.com>:
Hi,
I did some search about replacing osd, and found some different steps, probably for different release? Is there recommended process to replace an osd with Octopus? Two cases here: 1) replace HDD whose WAL and DB are on a SSD. 1-1) failed disk is replaced by the same model. 1-2) working disk is replaced by bigger one. 2) replace the SSD holding WAL and DB for multiple HDDs.
Thanks! Tony _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
When replacing an osd, there will be no PG remapping, and backfill will restore the data on the new disk, right?
That depends on how you decide to go through the replacement process. Usually without your intervention (e.g. setting the appropriate OSD flags) the remapping will happen after an OSD goes down and out.
This has been unclear to me. Is OSD going to be marked out and PGs going to be remapped during replacing? Or it depends on process? When mark an OSD out, remapping will happen and it will take some time for data migration. Is cluster in degraded state during such duration? My understanding is that, remapping only happens when the OSD is marked out. Replacement process will keep OSD always in, assuming replacing with the same disk model. In case to replace with different size, it could be more complicated, because weight has to be adjusted for size change and PG may be rebalanced.
In case of restoring a host with multiple OSDs, like WAL/DB SSD needs to be replaced, I see two options. 1) Keep cluster in degraded state and rebuild all OSDs. 2) Mark those OSDs out so PGs are rebalanced, rebuild OSDs, and bring them back in to rebalance PGs again.
These are basically your options, yes.
The key here is how much time backfilling and rebalancing will take? The intention is to not keep cluster in degraded state for too long. I assume they are similar, because either of them is to copy the same amount of data? If that's true, then option #2 is pointless. Could anyone share such experiences, like how long time it takes to recover how much data on what kind of networking/computing env?
No, option 2 is not pointless, it helps you prevent a degraded state. Having a small cluster or crush rules that only allow few failed OSDs it could be dangerous taking out an entire node, risking another failure and potential data loss. It highly depends on your specific setup and if you're willing to take the risk during rebuild of a node. The recovery/backfill speed is also depeneding on the size of OSDs, the object sizes, amount of data, etc. You would probably need to search the mailing list for examples from someone sharing their experience or so, I don't have captured such statistics.
My conclusion was based on two assumptions, correct me if they are wrong. 1) cluster is degraded during remapping. 2) no remapping when recovering an OSD. For option #1, no remapping, just degraded state during recovering. For option #2, remapping twice, one is to remap PG from old OSD to others, another is to remap PG again when new OSD is in place. It seems degrade state is twice longer with option #2 than #1. Is that right? Thanks! Tony
When replacing an osd, there will be no PG remapping, and backfill
will restore the data on the new disk, right?
That depends on how you decide to go through the replacement process. Usually without your intervention (e.g. setting the appropriate OSD flags) the remapping will happen after an OSD goes down and out.
This has been unclear to me. Is OSD going to be marked out and PGs going to be remapped during replacing? Or it depends on process?
When mark an OSD out, remapping will happen and it will take some time for data migration. Is cluster in degraded state during such duration?
If you set the `noout` flag on the affected OSDs or the entire cluster, there won’t be remapping. If the OSD fails and is marked `out`, there will be remapping and balancing.
My understanding is that, remapping only happens when the OSD is marked out.
CRUSH topology and rule changes can result in misplaced object too, but that’s a tangent.
Replacement process will keep OSD always in, assuming replacing with the same disk model.
`ceph osd destroy` is your friend.
In case to replace with different size, it could be more complicated, because weight has to be adjusted for size change and PG may be rebalanced.
If you replace an OSD with a drive of a different size, and you do so in a way such that the CRUSH weight is changed to match, then yes almost certainly some PG acting sets will change.
The key here is how much time backfilling and rebalancing will take? The intention is to not keep cluster in degraded state for too long. I assume they are similar, because either of them is to copy the same amount of data? If that's true, then option #2 is pointless. Could anyone share such experiences, like how long time it takes to recover how much data on what kind of networking/computing env?
No, option 2 is not pointless, it helps you prevent a degraded state. Having a small cluster or crush rules that only allow few failed OSDs it could be dangerous taking out an entire node, risking another failure and potential data loss. It highly depends on your specific setup and if you're willing to take the risk during rebuild of a node.
Agreed. Overlapping failures can and do happen. The flipside is that if one lets recovery complete, there has to be enough unused capacity in the right places to accomodate new data replicas. If, say, the affected cluster is on a different continent and you don’t have trustworthy 24x7 remote hands, then it could take some time to replace a failed drive or node. In this case, it likely is advantageous to let the cluster recover. If however you can get the affected drive / node back faster than recovery would take, it can be advantageous to prevent recovery until the OSDs are back up. Either way, Ceph has to create data replicas from survivors. *If* you can replace a drive immediately, then there’s no extra risk and you can cut data movement very roughly in half. This ties into the `mon_osd_down_out_subtree_limit` setting. Depending on one’s topology, it can prevent a thundering herd of recovery, with the idea that it’s often faster to get a node back up than it would be to recover all that data. This also avoids surviving OSDs potentially becoming full, but one has to have good monitoring so that this state does not continue indefinitely. Basically, any time PGs are undersized, there’s risk of an overlapping failure. The best course is often a question of which strategy will get them back to full size. Remapped PGs aren’t so big a deal, because at all times you have the desired number of replicas.
The recovery/backfill speed is also depeneding on the size of OSDs, the object sizes, amount of data, etc.
It’s also a function of HDD vs SSD, replication vs EC, whether omaps are significantly involved, throttle settings, cluster size and topology, etc.
You would probably need to search the mailing list for examples from someone sharing their experience or so, I don't have captured such statistics.
My conclusion was based on two assumptions, correct me if they are wrong. 1) cluster is degraded during remapping.
Be careful what you consider “degraded”.
2) no remapping when recovering an OSD.
For option #1, no remapping, just degraded state during recovering. For option #2, remapping twice, one is to remap PG from old OSD to others, another is to remap PG again when new OSD is in place. It seems degrade state is twice longer with option #2 than #1. Is that right?
It can be, depending on your topology.
When replacing an osd, there will be no PG remapping, and backfill
will restore the data on the new disk, right?
That depends on how you decide to go through the replacement process. Usually without your intervention (e.g. setting the appropriate OSD flags) the remapping will happen after an OSD goes down and out.
This has been unclear to me. Is OSD going to be marked out and PGs going to be remapped during replacing? Or it depends on process?
When mark an OSD out, remapping will happen and it will take some time for data migration. Is cluster in degraded state during such duration?
If you set the `noout` flag on the affected OSDs or the entire cluster, there won’t be remapping.
If the OSD fails and is marked `out`, there will be remapping and balancing.
Here is the context. https://docs.ceph.com/en/latest/mgr/orchestrator/#replace-an-osd When disk is broken, 1) orch osd rm <svc_id(s)> --replace [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file> Step #1 marks OSD "destroyed". I assume it has the same effect as "ceph osd destroy". And that keeps OSD "in", no PG remapping and cluster is in "degrade" state. After step #3, OSD will be "up" and "in", data will be recovered back to new disk. Is that right? Is cluster "degrade" or "healthy" during such recovery? For another option, the difference is no "--replace" in step #1. 1) orch osd rm <svc_id(s)> [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file> Step #1 evacuates PGs from OSD and removes it from cluster. If disk is broken or OSD daemon is down, is this evacuation still going to work? Is it going to take a while if there is lots data on this disk? After step #3, PGs will be rebalanced/remapped again when new OSD joins the cluster. I think, to replace with the same disk model, option #1 is preferred, to replace with different disk model, it needs to be option #2. Am I right? Any comments is welcome.
My understanding is that, remapping only happens when the OSD is marked out.
CRUSH topology and rule changes can result in misplaced object too, but that’s a tangent.
Replacement process will keep OSD always in, assuming replacing with the same disk model.
`ceph osd destroy` is your friend.
In case to replace with different size, it could be more complicated, because weight has to be adjusted for size change and PG may be rebalanced.
If you replace an OSD with a drive of a different size, and you do so in a way such that the CRUSH weight is changed to match, then yes almost certainly some PG acting sets will change.
The key here is how much time backfilling and rebalancing will take? The intention is to not keep cluster in degraded state for too long. I assume they are similar, because either of them is to copy the same amount of data? If that's true, then option #2 is pointless. Could anyone share such experiences, like how long time it takes to recover how much data on what kind of networking/computing env?
No, option 2 is not pointless, it helps you prevent a degraded state. Having a small cluster or crush rules that only allow few failed OSDs it could be dangerous taking out an entire node, risking another failure and potential data loss. It highly depends on your specific setup and if you're willing to take the risk during rebuild of a node.
Agreed. Overlapping failures can and do happen. The flipside is that if one lets recovery complete, there has to be enough unused capacity in the right places to accomodate new data replicas.
If, say, the affected cluster is on a different continent and you don’t have trustworthy 24x7 remote hands, then it could take some time to replace a failed drive or node. In this case, it likely is advantageous to let the cluster recover.
If however you can get the affected drive / node back faster than recovery would take, it can be advantageous to prevent recovery until the OSDs are back up. Either way, Ceph has to create data replicas from survivors. *If* you can replace a drive immediately, then there’s no extra risk and you can cut data movement very roughly in half.
This ties into the `mon_osd_down_out_subtree_limit` setting. Depending on one’s topology, it can prevent a thundering herd of recovery, with the idea that it’s often faster to get a node back up than it would be to recover all that data. This also avoids surviving OSDs potentially becoming full, but one has to have good monitoring so that this state does not continue indefinitely.
Basically, any time PGs are undersized, there’s risk of an overlapping failure. The best course is often a question of which strategy will get them back to full size. Remapped PGs aren’t so big a deal, because at all times you have the desired number of replicas.
The recovery/backfill speed is also depeneding on the size of OSDs, the object sizes, amount of data, etc.
It’s also a function of HDD vs SSD, replication vs EC, whether omaps are significantly involved, throttle settings, cluster size and topology, etc.
You would probably need to search the mailing list for examples from someone sharing their experience or so, I don't have captured such statistics.
My conclusion was based on two assumptions, correct me if they are wrong. 1) cluster is degraded during remapping.
Be careful what you consider “degraded”.
2) no remapping when recovering an OSD.
For option #1, no remapping, just degraded state during recovering. For option #2, remapping twice, one is to remap PG from old OSD to others, another is to remap PG again when new OSD is in place. It seems degrade state is twice longer with option #2 than #1. Is that right?
It can be, depending on your topology.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Here is the context. https://docs.ceph.com/en/latest/mgr/orchestrator/#replace-an-osd
When disk is broken, 1) orch osd rm <svc_id(s)> --replace [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file>
Step #1 marks OSD "destroyed". I assume it has the same effect as "ceph osd destroy". And that keeps OSD "in", no PG remapping and cluster is in "degrade" state.
After step #3, OSD will be "up" and "in", data will be recovered back to new disk. Is that right?
Yes.
Is cluster "degrade" or "healthy" during such recovery?
It will be degraded, because there are fewer copies of some data available than during normal operation. Clients will continue to access all data.
For another option, the difference is no "--replace" in step #1. 1) orch osd rm <svc_id(s)> [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file>
Step #1 evacuates PGs from OSD and removes it from cluster. If disk is broken or OSD daemon is down, is this evacuation still going to work?
Yes, of course — broken drives are the typical reason for removing OSDs.
Is it going to take a while if there is lots data on this disk?
Yes, depending on what “a while” means to you, the size of the cluster, whether the pool is replicated or EC, and whether these are HDDs or SSDs.
After step #3, PGs will be rebalanced/remapped again when new OSD joins the cluster.
I think, to replace with the same disk model, option #1 is preferred, to replace with different disk model, it needs to be option #2.
I haven’t tried it under Octopus, but I don’t think this is strictly true. If you replace it with a different model that is approximately the same size, everything will be fine. Through Luminous and I think Nautilus at least, if you `destroy` and replace with a larger drive, the CRUSH weight of the OSD will still reflect that of the old drive. You could then run `ceph osd crush reweight` after deploying to adjust the size. You could record the CRUSH weights of all your drive models for initial OSD deploys, or you could `ceph osd tree` and look for another OSD of the same model, and set the CRUSH weight accordingly. If you replace with a smaller drive, your cluster will lose a small amount of usable capacity. If you replace with a larger drive, the cluster may or may not enjoy a slight increase in capacity — that depends on replication strategy, rack/host weights, etc. My personal philosophy on drive replacements: o Build OSDs with `—dmcrypt` so that you don’t have to worry about data if/when you RMA or recycle bad drives. RMAs are a hassle, so pick a certain value threshold before a drive is worth the effort. This might be in the $250-500 range for example, which means that for many HDDs it isn’t worth RMAing them. o If you have an exact replacement, use it o When buying spares, buy the largest size drive you have deployed — or will deploy within the next year or so. That way you know that your spares can take the place of any drive you have, so you don’t have to maintain stock of more than one size. Worst case you don’t immediately make good use of that extra capacity, but you may in the future as drives in other failure domains fail and are replaced. Be careful, though of mixing drives that a lot different in size. Mixing 12 and 14 TB drives, even 12 and 16 is usually no big deal, but if you mix say 1TB and 16 TB drives, you can end up exceeding `mon_max_pg_per_osd`. Which is one reason why I like to increase it from the default value to, say, 400.
Hi all, maybe a further alternative. With our support contract I get exact replacements. I found out that doing an off-line copy of a still readable OSD with ddrescue speeds things up dramatically and avoids extended periods of degraded PGs. Situation and what I did: I had a disk with repeated deep scrub errors and checking with smartctl I could see that it started remapping sectors. This showed up as PG scrub error. I initiated a full deep scrub of the disk and run PG repair on every PG that was marked as having errors. This way, ceph rewrites the broken object and the disk writes it to a remapped, that is, healthy sector. Doing this a couple of times will leave you with a disk that is 100% readable. I then shut the OSD down. This lead to recovery IO as expected and after less than 2 hours everything was rebuilt to full redundancy (it was probably faster, I only checked after 2 hours). Recovery from single disk fail is very fast due to all-to-all rebuild. In the mean time, I did a full disk copy with ddrescue to a large file system space I have on a copy station. Took 16h for a 12TB drive. Right after this, the replacement arrived and I copied the image back. Another 16h. After this, I simply inserted the new disk with the 5 days old OSD copy and brought it up (there was a weekend in between). Almost all objects on the drive were still up-to-date and after just 30 minutes all PGs were active and clean. Nothing remapped or misplaced any more. For comparison, I once added a single drive and it took 2 weeks for the affected PGs to be active+clean again. The off-line copy can use much more aggressive and effective IO to a single drive than ceph rebalancing ever would. For single-disk exchange on our service contract I will probably continue with the ddrescue method even though it requires manual action. For the future I plan to adapt a different strategy to utilize the all-to-all copy capability of ceph. Exchanging single disks seems not to be a good way to run ceph. I will rather have a larger amount of disks act as hot spares. For example, having enough capacity that one can tolerate loosing 10% of all disks before replacing anything. Adding a large number of disks is overall more effective as it will basically take the same time to get back to health OK as exchanging a single disk. With my timings, this "replace many disks not single ones" will amortise if at least 5-6 drives failed and are down+out. It will also limit writes to degraded PGs to the shortest interval possible. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: 28 November 2020 05:55:06 To: Tony Liu Cc: ceph-users@ceph.io Subject: [ceph-users] Re: replace osd with Octopus
Here is the context. https://docs.ceph.com/en/latest/mgr/orchestrator/#replace-an-osd
When disk is broken, 1) orch osd rm <svc_id(s)> --replace [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file>
Step #1 marks OSD "destroyed". I assume it has the same effect as "ceph osd destroy". And that keeps OSD "in", no PG remapping and cluster is in "degrade" state.
After step #3, OSD will be "up" and "in", data will be recovered back to new disk. Is that right?
Yes.
Is cluster "degrade" or "healthy" during such recovery?
It will be degraded, because there are fewer copies of some data available than during normal operation. Clients will continue to access all data.
For another option, the difference is no "--replace" in step #1. 1) orch osd rm <svc_id(s)> [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file>
Step #1 evacuates PGs from OSD and removes it from cluster. If disk is broken or OSD daemon is down, is this evacuation still going to work?
Yes, of course — broken drives are the typical reason for removing OSDs.
Is it going to take a while if there is lots data on this disk?
Yes, depending on what “a while” means to you, the size of the cluster, whether the pool is replicated or EC, and whether these are HDDs or SSDs.
After step #3, PGs will be rebalanced/remapped again when new OSD joins the cluster.
I think, to replace with the same disk model, option #1 is preferred, to replace with different disk model, it needs to be option #2.
I haven’t tried it under Octopus, but I don’t think this is strictly true. If you replace it with a different model that is approximately the same size, everything will be fine. Through Luminous and I think Nautilus at least, if you `destroy` and replace with a larger drive, the CRUSH weight of the OSD will still reflect that of the old drive. You could then run `ceph osd crush reweight` after deploying to adjust the size. You could record the CRUSH weights of all your drive models for initial OSD deploys, or you could `ceph osd tree` and look for another OSD of the same model, and set the CRUSH weight accordingly. If you replace with a smaller drive, your cluster will lose a small amount of usable capacity. If you replace with a larger drive, the cluster may or may not enjoy a slight increase in capacity — that depends on replication strategy, rack/host weights, etc. My personal philosophy on drive replacements: o Build OSDs with `—dmcrypt` so that you don’t have to worry about data if/when you RMA or recycle bad drives. RMAs are a hassle, so pick a certain value threshold before a drive is worth the effort. This might be in the $250-500 range for example, which means that for many HDDs it isn’t worth RMAing them. o If you have an exact replacement, use it o When buying spares, buy the largest size drive you have deployed — or will deploy within the next year or so. That way you know that your spares can take the place of any drive you have, so you don’t have to maintain stock of more than one size. Worst case you don’t immediately make good use of that extra capacity, but you may in the future as drives in other failure domains fail and are replaced. Be careful, though of mixing drives that a lot different in size. Mixing 12 and 14 TB drives, even 12 and 16 is usually no big deal, but if you mix say 1TB and 16 TB drives, you can end up exceeding `mon_max_pg_per_osd`. Which is one reason why I like to increase it from the default value to, say, 400. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Frank, A dummy question, what's this all-to-all rebuild/copy? Is that PG remapping when the broken disk is taken out? In your case, does "shut the OSD down" mark OSD "out"? "rebuilt to full redundancy" took 2 hours (I assume there was PG remapping.)? What's the disk size? Regarding to your future plan relying on all-to-all copy, "with large amount of hot spares", I assume you mean large amount of spare spaces? What do you do when a disk fails? Just take it out and let the cluster heal itself by remapping PGs from failed disk to spare spaces? Thanks! Tony
-----Original Message----- From: Frank Schilder <frans@dtu.dk> Sent: Saturday, November 28, 2020 12:42 AM To: Anthony D'Atri <anthony.datri@gmail.com>; Tony Liu <tonyliu0592@hotmail.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: replace osd with Octopus
Hi all,
maybe a further alternative.
With our support contract I get exact replacements. I found out that doing an off-line copy of a still readable OSD with ddrescue speeds things up dramatically and avoids extended periods of degraded PGs.
Situation and what I did:
I had a disk with repeated deep scrub errors and checking with smartctl I could see that it started remapping sectors. This showed up as PG scrub error. I initiated a full deep scrub of the disk and run PG repair on every PG that was marked as having errors. This way, ceph rewrites the broken object and the disk writes it to a remapped, that is, healthy sector. Doing this a couple of times will leave you with a disk that is 100% readable.
I then shut the OSD down. This lead to recovery IO as expected and after less than 2 hours everything was rebuilt to full redundancy (it was probably faster, I only checked after 2 hours). Recovery from single disk fail is very fast due to all-to-all rebuild.
In the mean time, I did a full disk copy with ddrescue to a large file system space I have on a copy station. Took 16h for a 12TB drive. Right after this, the replacement arrived and I copied the image back. Another 16h.
After this, I simply inserted the new disk with the 5 days old OSD copy and brought it up (there was a weekend in between). Almost all objects on the drive were still up-to-date and after just 30 minutes all PGs were active and clean. Nothing remapped or misplaced any more.
For comparison, I once added a single drive and it took 2 weeks for the affected PGs to be active+clean again. The off-line copy can use much more aggressive and effective IO to a single drive than ceph rebalancing ever would.
For single-disk exchange on our service contract I will probably continue with the ddrescue method even though it requires manual action.
For the future I plan to adapt a different strategy to utilize the all- to-all copy capability of ceph. Exchanging single disks seems not to be a good way to run ceph. I will rather have a larger amount of disks act as hot spares. For example, having enough capacity that one can tolerate loosing 10% of all disks before replacing anything. Adding a large number of disks is overall more effective as it will basically take the same time to get back to health OK as exchanging a single disk.
With my timings, this "replace many disks not single ones" will amortise if at least 5-6 drives failed and are down+out. It will also limit writes to degraded PGs to the shortest interval possible.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: 28 November 2020 05:55:06 To: Tony Liu Cc: ceph-users@ceph.io Subject: [ceph-users] Re: replace osd with Octopus
Here is the context. https://docs.ceph.com/en/latest/mgr/orchestrator/#replace-an-osd
When disk is broken, 1) orch osd rm <svc_id(s)> --replace [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file>
Step #1 marks OSD "destroyed". I assume it has the same effect as "ceph osd destroy". And that keeps OSD "in", no PG remapping and cluster is in "degrade" state.
After step #3, OSD will be "up" and "in", data will be recovered back to new disk. Is that right?
Yes.
Is cluster "degrade" or "healthy" during such recovery?
It will be degraded, because there are fewer copies of some data available than during normal operation. Clients will continue to access all data.
For another option, the difference is no "--replace" in step #1. 1) orch osd rm <svc_id(s)> [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file>
Step #1 evacuates PGs from OSD and removes it from cluster. If disk is broken or OSD daemon is down, is this evacuation still going to work?
Yes, of course - broken drives are the typical reason for removing OSDs.
Is it going to take a while if there is lots data on this disk?
Yes, depending on what "a while" means to you, the size of the cluster, whether the pool is replicated or EC, and whether these are HDDs or SSDs.
After step #3, PGs will be rebalanced/remapped again when new OSD joins the cluster.
I think, to replace with the same disk model, option #1 is preferred, to replace with different disk model, it needs to be option #2.
I haven't tried it under Octopus, but I don't think this is strictly true. If you replace it with a different model that is approximately the same size, everything will be fine. Through Luminous and I think Nautilus at least, if you `destroy` and replace with a larger drive, the CRUSH weight of the OSD will still reflect that of the old drive. You could then run `ceph osd crush reweight` after deploying to adjust the size. You could record the CRUSH weights of all your drive models for initial OSD deploys, or you could `ceph osd tree` and look for another OSD of the same model, and set the CRUSH weight accordingly.
If you replace with a smaller drive, your cluster will lose a small amount of usable capacity. If you replace with a larger drive, the cluster may or may not enjoy a slight increase in capacity - that depends on replication strategy, rack/host weights, etc.
My personal philosophy on drive replacements:
o Build OSDs with `-dmcrypt` so that you don't have to worry about data if/when you RMA or recycle bad drives. RMAs are a hassle, so pick a certain value threshold before a drive is worth the effort. This might be in the $250-500 range for example, which means that for many HDDs it isn't worth RMAing them.
o If you have an exact replacement, use it
o When buying spares, buy the largest size drive you have deployed - or will deploy within the next year or so. That way you know that your spares can take the place of any drive you have, so you don't have to maintain stock of more than one size. Worst case you don't immediately make good use of that extra capacity, but you may in the future as drives in other failure domains fail and are replaced. Be careful, though of mixing drives that a lot different in size. Mixing 12 and 14 TB drives, even 12 and 16 is usually no big deal, but if you mix say 1TB and 16 TB drives, you can end up exceeding `mon_max_pg_per_osd`. Which is one reason why I like to increase it from the default value to, say, 400. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
A dummy question, what's this all-to-all rebuild/copy? Is that PG remapping when the broken disk is taken out?
- all-to-all: every OSD sends/receives objects to/from every other OSD - one-to-all: one OSD sends objects to all other OSDs - all-to-one: all other OSDs send objects to one OSD All-to-all happens if one disk fails and all other OSDs rebuild the missing data. This is very fast. One-to-all happens when you evacuate a single disk, for example, by setting its weight to 0. This is very slow. It is faster to just fail the disk and let the data rebuild, however, with the drawback of temporarily reduced redundancy. All-to-one happens when you add a single disk and all other OSDs send it its data. This is also very slow and there is no short-cut. Conclusion: design work flows that utilize the all-to-all capability of ceph as much as possible. For example, plan the cluster such that single-disk operations can be avoided.
In your case, does "shut the OSD down" mark OSD "out"? "rebuilt to full redundancy" took 2 hours (I assume there was PG remapping.)? What's the disk size?
If you stop an OSD, it will be down and 5 minutes later marked out (auto-out). These time-outs can be configured. Size was 12TB (10.7TiB). Its NL-SAS drives.
Regarding to your future plan relying on all-to-all copy, "with large amount of hot spares", I assume you mean large amount of spare spaces? What do you do when a disk fails? Just take it out and let the cluster heal itself by remapping PGs from failed disk to spare spaces?
Hot spares means that you deploy 5-10% more disks than you need to provide the requested capacity (hot means they are already part of the cluster, otherwise they would be called cold spares). Then, if a single disk fails, you do nothing, because you still have excess capacity. Only after all the 5-10% extra disks have failed will 5-10% disks be added again as new. In fact, I would plan it such that this replacement falls together with the next capacity extension. Then, you simply do nothing when a disk fails - except maybe taking it out and requesting a replacement if your contract provides that (put it on a shelf until next cluster extension). Doubling the number of OSDs in a storage extension operation will practically result in all-to-all data movement. Its theoretically half-to-half, but more than 50% of objects are usually misplaced and there will be movement between the original set of OSDs as well. In any case, getting such a large number of disks involved that only need to be filled up to 50% of the previous capacity will be much more efficient (in administrator workload/salary) than doing single-disk replacements or tiny extensions. Ceph is fun if its big enough :) Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: 02 December 2020 05:48:10 To: Frank Schilder; Anthony D'Atri Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: replace osd with Octopus Hi Frank, A dummy question, what's this all-to-all rebuild/copy? Is that PG remapping when the broken disk is taken out? In your case, does "shut the OSD down" mark OSD "out"? "rebuilt to full redundancy" took 2 hours (I assume there was PG remapping.)? What's the disk size? Regarding to your future plan relying on all-to-all copy, "with large amount of hot spares", I assume you mean large amount of spare spaces? What do you do when a disk fails? Just take it out and let the cluster heal itself by remapping PGs from failed disk to spare spaces? Thanks! Tony
-----Original Message----- From: Frank Schilder <frans@dtu.dk> Sent: Saturday, November 28, 2020 12:42 AM To: Anthony D'Atri <anthony.datri@gmail.com>; Tony Liu <tonyliu0592@hotmail.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: replace osd with Octopus
Hi all,
maybe a further alternative.
With our support contract I get exact replacements. I found out that doing an off-line copy of a still readable OSD with ddrescue speeds things up dramatically and avoids extended periods of degraded PGs.
Situation and what I did:
I had a disk with repeated deep scrub errors and checking with smartctl I could see that it started remapping sectors. This showed up as PG scrub error. I initiated a full deep scrub of the disk and run PG repair on every PG that was marked as having errors. This way, ceph rewrites the broken object and the disk writes it to a remapped, that is, healthy sector. Doing this a couple of times will leave you with a disk that is 100% readable.
I then shut the OSD down. This lead to recovery IO as expected and after less than 2 hours everything was rebuilt to full redundancy (it was probably faster, I only checked after 2 hours). Recovery from single disk fail is very fast due to all-to-all rebuild.
In the mean time, I did a full disk copy with ddrescue to a large file system space I have on a copy station. Took 16h for a 12TB drive. Right after this, the replacement arrived and I copied the image back. Another 16h.
After this, I simply inserted the new disk with the 5 days old OSD copy and brought it up (there was a weekend in between). Almost all objects on the drive were still up-to-date and after just 30 minutes all PGs were active and clean. Nothing remapped or misplaced any more.
For comparison, I once added a single drive and it took 2 weeks for the affected PGs to be active+clean again. The off-line copy can use much more aggressive and effective IO to a single drive than ceph rebalancing ever would.
For single-disk exchange on our service contract I will probably continue with the ddrescue method even though it requires manual action.
For the future I plan to adapt a different strategy to utilize the all- to-all copy capability of ceph. Exchanging single disks seems not to be a good way to run ceph. I will rather have a larger amount of disks act as hot spares. For example, having enough capacity that one can tolerate loosing 10% of all disks before replacing anything. Adding a large number of disks is overall more effective as it will basically take the same time to get back to health OK as exchanging a single disk.
With my timings, this "replace many disks not single ones" will amortise if at least 5-6 drives failed and are down+out. It will also limit writes to degraded PGs to the shortest interval possible.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: 28 November 2020 05:55:06 To: Tony Liu Cc: ceph-users@ceph.io Subject: [ceph-users] Re: replace osd with Octopus
Here is the context. https://docs.ceph.com/en/latest/mgr/orchestrator/#replace-an-osd
When disk is broken, 1) orch osd rm <svc_id(s)> --replace [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file>
Step #1 marks OSD "destroyed". I assume it has the same effect as "ceph osd destroy". And that keeps OSD "in", no PG remapping and cluster is in "degrade" state.
After step #3, OSD will be "up" and "in", data will be recovered back to new disk. Is that right?
Yes.
Is cluster "degrade" or "healthy" during such recovery?
It will be degraded, because there are fewer copies of some data available than during normal operation. Clients will continue to access all data.
For another option, the difference is no "--replace" in step #1. 1) orch osd rm <svc_id(s)> [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file>
Step #1 evacuates PGs from OSD and removes it from cluster. If disk is broken or OSD daemon is down, is this evacuation still going to work?
Yes, of course - broken drives are the typical reason for removing OSDs.
Is it going to take a while if there is lots data on this disk?
Yes, depending on what "a while" means to you, the size of the cluster, whether the pool is replicated or EC, and whether these are HDDs or SSDs.
After step #3, PGs will be rebalanced/remapped again when new OSD joins the cluster.
I think, to replace with the same disk model, option #1 is preferred, to replace with different disk model, it needs to be option #2.
I haven't tried it under Octopus, but I don't think this is strictly true. If you replace it with a different model that is approximately the same size, everything will be fine. Through Luminous and I think Nautilus at least, if you `destroy` and replace with a larger drive, the CRUSH weight of the OSD will still reflect that of the old drive. You could then run `ceph osd crush reweight` after deploying to adjust the size. You could record the CRUSH weights of all your drive models for initial OSD deploys, or you could `ceph osd tree` and look for another OSD of the same model, and set the CRUSH weight accordingly.
If you replace with a smaller drive, your cluster will lose a small amount of usable capacity. If you replace with a larger drive, the cluster may or may not enjoy a slight increase in capacity - that depends on replication strategy, rack/host weights, etc.
My personal philosophy on drive replacements:
o Build OSDs with `-dmcrypt` so that you don't have to worry about data if/when you RMA or recycle bad drives. RMAs are a hassle, so pick a certain value threshold before a drive is worth the effort. This might be in the $250-500 range for example, which means that for many HDDs it isn't worth RMAing them.
o If you have an exact replacement, use it
o When buying spares, buy the largest size drive you have deployed - or will deploy within the next year or so. That way you know that your spares can take the place of any drive you have, so you don't have to maintain stock of more than one size. Worst case you don't immediately make good use of that extra capacity, but you may in the future as drives in other failure domains fail and are replaced. Be careful, though of mixing drives that a lot different in size. Mixing 12 and 14 TB drives, even 12 and 16 is usually no big deal, but if you mix say 1TB and 16 TB drives, you can end up exceeding `mon_max_pg_per_osd`. Which is one reason why I like to increase it from the default value to, say, 400. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
A dummy question, what's this all-to-all rebuild/copy? Is that PG remapping when the broken disk is taken out?
- all-to-all: every OSD sends/receives objects to/from every other OSD - one-to-all: one OSD sends objects to all other OSDs - all-to-one: all other OSDs send objects to one OSD
All-to-all happens if one disk fails and all other OSDs rebuild the missing data. This is very fast.
No matter "up" or "down", PG mapping remains when OSD is "in", PGs will be remapped when OSD is "out". Is that correct? Since "other OSDs rebuild the missing data", there must be PG remapping and the failed disk is "out" by either manual or automatic. Right?
One-to-all happens when you evacuate a single disk, for example, by setting its weight to 0. This is very slow. It is faster to just fail the disk and let the data rebuild, however, with the drawback of temporarily reduced redundancy.
From what I see, the difference between "evacuate a single disk" and "a disk fails" is cluster state. When "evacuate a single disk", cluster is healthy because all replicas are available. When "a disk fails", cluster is degraded, because one replica is missing. In terms of PG remapping, it happens either way. I see the same copy happening in background, PGs on failed/evacuated disk are copied to other disks. If that's true, why there is dramatic timing difference for those two cases? Give my above understanding, all-to-all is no difference from one-to-all. In either case, PGs of one disk are remapped to others. I must be missing something seriously:)
All-to-one happens when you add a single disk and all other OSDs send it its data. This is also very slow and there is no short-cut.
Add a new disk will cause PGs to be rebalanced. It will take times. But for replacing disk (OSD keeps being "in".), since PG mapping remains, no rebalance/remapping, just copy data back.
Conclusion: design work flows that utilize the all-to-all capability of ceph as much as possible. For example, plan the cluster such that single-disk operations can be avoided.
In your case, does "shut the OSD down" mark OSD "out"? "rebuilt to full redundancy" took 2 hours (I assume there was PG remapping.)? What's the disk size?
If you stop an OSD, it will be down and 5 minutes later marked out (auto-out). These time-outs can be configured. Size was 12TB (10.7TiB). Its NL-SAS drives.
Regarding to your future plan relying on all-to-all copy, "with large amount of hot spares", I assume you mean large amount of spare spaces? What do you do when a disk fails? Just take it out and let the cluster heal itself by remapping PGs from failed disk to spare spaces?
Hot spares means that you deploy 5-10% more disks than you need to provide the requested capacity (hot means they are already part of the cluster, otherwise they would be called cold spares). Then, if a single disk fails, you do nothing, because you still have excess capacity. Only after all the 5-10% extra disks have failed will 5-10% disks be added again as new. In fact, I would plan it such that this replacement falls together with the next capacity extension. Then, you simply do nothing when a disk fails - except maybe taking it out and requesting a replacement if your contract provides that (put it on a shelf until next cluster extension).
Is hot spare disk "in" the cluster and allocated with PGs? If yes, what's the difference between hot spare disk and normal disks? My understanding is that, just keep cluster capacity under a reasonable threshold, to accommodate one or couple disks failure. Since the cluster will heal itself, no rush to replace the disk when failure happens. And when replacing the disk, it will be the same as adding a new disk. This is my original option #2. I was just not sure about how much time the cluster will take to heal itself. Based on your experiences, it's pretty fast, couple hours to rebuild 10T data.
Doubling the number of OSDs in a storage extension operation will practically result in all-to-all data movement. Its theoretically half- to-half, but more than 50% of objects are usually misplaced and there will be movement between the original set of OSDs as well. In any case, getting such a large number of disks involved that only need to be filled up to 50% of the previous capacity will be much more efficient (in administrator workload/salary) than doing single-disk replacements or tiny extensions.
Ceph is fun if its big enough :)
Definitely!
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: 02 December 2020 05:48:10 To: Frank Schilder; Anthony D'Atri Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: replace osd with Octopus
Hi Frank,
A dummy question, what's this all-to-all rebuild/copy? Is that PG remapping when the broken disk is taken out?
In your case, does "shut the OSD down" mark OSD "out"? "rebuilt to full redundancy" took 2 hours (I assume there was PG remapping.)? What's the disk size?
Regarding to your future plan relying on all-to-all copy, "with large amount of hot spares", I assume you mean large amount of spare spaces? What do you do when a disk fails? Just take it out and let the cluster heal itself by remapping PGs from failed disk to spare spaces?
-----Original Message----- From: Frank Schilder <frans@dtu.dk> Sent: Saturday, November 28, 2020 12:42 AM To: Anthony D'Atri <anthony.datri@gmail.com>; Tony Liu <tonyliu0592@hotmail.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: replace osd with Octopus
Hi all,
maybe a further alternative.
With our support contract I get exact replacements. I found out that doing an off-line copy of a still readable OSD with ddrescue speeds things up dramatically and avoids extended periods of degraded PGs.
Situation and what I did:
I had a disk with repeated deep scrub errors and checking with smartctl I could see that it started remapping sectors. This showed up as PG scrub error. I initiated a full deep scrub of the disk and run PG repair on every PG that was marked as having errors. This way, ceph rewrites the broken object and the disk writes it to a remapped, that is, healthy sector. Doing this a couple of times will leave you with a disk that is 100% readable.
I then shut the OSD down. This lead to recovery IO as expected and after less than 2 hours everything was rebuilt to full redundancy (it was probably faster, I only checked after 2 hours). Recovery from single disk fail is very fast due to all-to-all rebuild.
In the mean time, I did a full disk copy with ddrescue to a large file system space I have on a copy station. Took 16h for a 12TB drive. Right after this, the replacement arrived and I copied the image back. Another 16h.
After this, I simply inserted the new disk with the 5 days old OSD copy and brought it up (there was a weekend in between). Almost all objects on the drive were still up-to-date and after just 30 minutes all PGs were active and clean. Nothing remapped or misplaced any more.
For comparison, I once added a single drive and it took 2 weeks for the affected PGs to be active+clean again. The off-line copy can use much more aggressive and effective IO to a single drive than ceph rebalancing ever would.
For single-disk exchange on our service contract I will probably continue with the ddrescue method even though it requires manual action.
For the future I plan to adapt a different strategy to utilize the all- to-all copy capability of ceph. Exchanging single disks seems not to be a good way to run ceph. I will rather have a larger amount of disks act as hot spares. For example, having enough capacity that one can tolerate loosing 10% of all disks before replacing anything. Adding a large number of disks is overall more effective as it will basically take the same time to get back to health OK as exchanging a single disk.
With my timings, this "replace many disks not single ones" will amortise if at least 5-6 drives failed and are down+out. It will also limit writes to degraded PGs to the shortest interval possible.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: 28 November 2020 05:55:06 To: Tony Liu Cc: ceph-users@ceph.io Subject: [ceph-users] Re: replace osd with Octopus
Here is the context. https://docs.ceph.com/en/latest/mgr/orchestrator/#replace-an-osd
When disk is broken, 1) orch osd rm <svc_id(s)> --replace [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file>
Step #1 marks OSD "destroyed". I assume it has the same effect as "ceph osd destroy". And that keeps OSD "in", no PG remapping and cluster is in "degrade" state.
After step #3, OSD will be "up" and "in", data will be recovered back to new disk. Is that right?
Yes.
Is cluster "degrade" or "healthy" during such recovery?
It will be degraded, because there are fewer copies of some data available than during normal operation. Clients will continue to access all data.
For another option, the difference is no "--replace" in step #1. 1) orch osd rm <svc_id(s)> [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file>
Step #1 evacuates PGs from OSD and removes it from cluster. If disk is broken or OSD daemon is down, is this evacuation still going to work?
Yes, of course - broken drives are the typical reason for removing OSDs.
Is it going to take a while if there is lots data on this disk?
Yes, depending on what "a while" means to you, the size of the cluster, whether the pool is replicated or EC, and whether these are HDDs or SSDs.
After step #3, PGs will be rebalanced/remapped again when new OSD joins the cluster.
I think, to replace with the same disk model, option #1 is preferred, to replace with different disk model, it needs to be
Thanks! Tony option #2.
I haven't tried it under Octopus, but I don't think this is strictly true. If you replace it with a different model that is approximately the same size, everything will be fine. Through Luminous and I think Nautilus at least, if you `destroy` and replace with a larger drive, the CRUSH weight of the OSD will still reflect that of the old drive. You could then run `ceph osd crush reweight` after deploying to adjust the size. You could record the CRUSH weights of all your drive models for initial OSD deploys, or you could `ceph osd tree` and look for another OSD of the same model, and set the CRUSH weight accordingly.
If you replace with a smaller drive, your cluster will lose a small amount of usable capacity. If you replace with a larger drive, the cluster may or may not enjoy a slight increase in capacity - that depends on replication strategy, rack/host weights, etc.
My personal philosophy on drive replacements:
o Build OSDs with `-dmcrypt` so that you don't have to worry about data if/when you RMA or recycle bad drives. RMAs are a hassle, so pick a certain value threshold before a drive is worth the effort. This might be in the $250-500 range for example, which means that for many HDDs it isn't worth RMAing them.
o If you have an exact replacement, use it
o When buying spares, buy the largest size drive you have deployed - or will deploy within the next year or so. That way you know that your spares can take the place of any drive you have, so you don't have to maintain stock of more than one size. Worst case you don't immediately make good use of that extra capacity, but you may in the future as drives in other failure domains fail and are replaced. Be careful, though of mixing drives that a lot different in size. Mixing 12 and 14 TB drives, even 12 and 16 is usually no big deal, but if you mix say 1TB and 16 TB drives, you can end up exceeding `mon_max_pg_per_osd`. Which is one reason why I like to increase it from the default value to, say, 400. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I must be missing something seriously:)
Yes. And I think its time that you actually try it out instead of writing ever longer e-mails. If you re-read the e-mail correspondence carefully, you should notice that your follow-up questions have been answered already. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: 02 December 2020 19:00:18 To: Frank Schilder; Anthony D'Atri Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: replace osd with Octopus
A dummy question, what's this all-to-all rebuild/copy? Is that PG remapping when the broken disk is taken out?
- all-to-all: every OSD sends/receives objects to/from every other OSD - one-to-all: one OSD sends objects to all other OSDs - all-to-one: all other OSDs send objects to one OSD
All-to-all happens if one disk fails and all other OSDs rebuild the missing data. This is very fast.
No matter "up" or "down", PG mapping remains when OSD is "in", PGs will be remapped when OSD is "out". Is that correct? Since "other OSDs rebuild the missing data", there must be PG remapping and the failed disk is "out" by either manual or automatic. Right?
One-to-all happens when you evacuate a single disk, for example, by setting its weight to 0. This is very slow. It is faster to just fail the disk and let the data rebuild, however, with the drawback of temporarily reduced redundancy.
From what I see, the difference between "evacuate a single disk" and "a disk fails" is cluster state. When "evacuate a single disk", cluster is healthy because all replicas are available. When "a disk fails", cluster is degraded, because one replica is missing. In terms of PG remapping, it happens either way. I see the same copy happening in background, PGs on failed/evacuated disk are copied to other disks. If that's true, why there is dramatic timing difference for those two cases? Give my above understanding, all-to-all is no difference from one-to-all. In either case, PGs of one disk are remapped to others. I must be missing something seriously:)
All-to-one happens when you add a single disk and all other OSDs send it its data. This is also very slow and there is no short-cut.
Add a new disk will cause PGs to be rebalanced. It will take times. But for replacing disk (OSD keeps being "in".), since PG mapping remains, no rebalance/remapping, just copy data back.
Conclusion: design work flows that utilize the all-to-all capability of ceph as much as possible. For example, plan the cluster such that single-disk operations can be avoided.
In your case, does "shut the OSD down" mark OSD "out"? "rebuilt to full redundancy" took 2 hours (I assume there was PG remapping.)? What's the disk size?
If you stop an OSD, it will be down and 5 minutes later marked out (auto-out). These time-outs can be configured. Size was 12TB (10.7TiB). Its NL-SAS drives.
Regarding to your future plan relying on all-to-all copy, "with large amount of hot spares", I assume you mean large amount of spare spaces? What do you do when a disk fails? Just take it out and let the cluster heal itself by remapping PGs from failed disk to spare spaces?
Hot spares means that you deploy 5-10% more disks than you need to provide the requested capacity (hot means they are already part of the cluster, otherwise they would be called cold spares). Then, if a single disk fails, you do nothing, because you still have excess capacity. Only after all the 5-10% extra disks have failed will 5-10% disks be added again as new. In fact, I would plan it such that this replacement falls together with the next capacity extension. Then, you simply do nothing when a disk fails - except maybe taking it out and requesting a replacement if your contract provides that (put it on a shelf until next cluster extension).
Is hot spare disk "in" the cluster and allocated with PGs? If yes, what's the difference between hot spare disk and normal disks? My understanding is that, just keep cluster capacity under a reasonable threshold, to accommodate one or couple disks failure. Since the cluster will heal itself, no rush to replace the disk when failure happens. And when replacing the disk, it will be the same as adding a new disk. This is my original option #2. I was just not sure about how much time the cluster will take to heal itself. Based on your experiences, it's pretty fast, couple hours to rebuild 10T data.
Doubling the number of OSDs in a storage extension operation will practically result in all-to-all data movement. Its theoretically half- to-half, but more than 50% of objects are usually misplaced and there will be movement between the original set of OSDs as well. In any case, getting such a large number of disks involved that only need to be filled up to 50% of the previous capacity will be much more efficient (in administrator workload/salary) than doing single-disk replacements or tiny extensions.
Ceph is fun if its big enough :)
Definitely!
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Tony Liu <tonyliu0592@hotmail.com> Sent: 02 December 2020 05:48:10 To: Frank Schilder; Anthony D'Atri Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: replace osd with Octopus
Hi Frank,
A dummy question, what's this all-to-all rebuild/copy? Is that PG remapping when the broken disk is taken out?
In your case, does "shut the OSD down" mark OSD "out"? "rebuilt to full redundancy" took 2 hours (I assume there was PG remapping.)? What's the disk size?
Regarding to your future plan relying on all-to-all copy, "with large amount of hot spares", I assume you mean large amount of spare spaces? What do you do when a disk fails? Just take it out and let the cluster heal itself by remapping PGs from failed disk to spare spaces?
-----Original Message----- From: Frank Schilder <frans@dtu.dk> Sent: Saturday, November 28, 2020 12:42 AM To: Anthony D'Atri <anthony.datri@gmail.com>; Tony Liu <tonyliu0592@hotmail.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: replace osd with Octopus
Hi all,
maybe a further alternative.
With our support contract I get exact replacements. I found out that doing an off-line copy of a still readable OSD with ddrescue speeds things up dramatically and avoids extended periods of degraded PGs.
Situation and what I did:
I had a disk with repeated deep scrub errors and checking with smartctl I could see that it started remapping sectors. This showed up as PG scrub error. I initiated a full deep scrub of the disk and run PG repair on every PG that was marked as having errors. This way, ceph rewrites the broken object and the disk writes it to a remapped, that is, healthy sector. Doing this a couple of times will leave you with a disk that is 100% readable.
I then shut the OSD down. This lead to recovery IO as expected and after less than 2 hours everything was rebuilt to full redundancy (it was probably faster, I only checked after 2 hours). Recovery from single disk fail is very fast due to all-to-all rebuild.
In the mean time, I did a full disk copy with ddrescue to a large file system space I have on a copy station. Took 16h for a 12TB drive. Right after this, the replacement arrived and I copied the image back. Another 16h.
After this, I simply inserted the new disk with the 5 days old OSD copy and brought it up (there was a weekend in between). Almost all objects on the drive were still up-to-date and after just 30 minutes all PGs were active and clean. Nothing remapped or misplaced any more.
For comparison, I once added a single drive and it took 2 weeks for the affected PGs to be active+clean again. The off-line copy can use much more aggressive and effective IO to a single drive than ceph rebalancing ever would.
For single-disk exchange on our service contract I will probably continue with the ddrescue method even though it requires manual action.
For the future I plan to adapt a different strategy to utilize the all- to-all copy capability of ceph. Exchanging single disks seems not to be a good way to run ceph. I will rather have a larger amount of disks act as hot spares. For example, having enough capacity that one can tolerate loosing 10% of all disks before replacing anything. Adding a large number of disks is overall more effective as it will basically take the same time to get back to health OK as exchanging a single disk.
With my timings, this "replace many disks not single ones" will amortise if at least 5-6 drives failed and are down+out. It will also limit writes to degraded PGs to the shortest interval possible.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Anthony D'Atri <anthony.datri@gmail.com> Sent: 28 November 2020 05:55:06 To: Tony Liu Cc: ceph-users@ceph.io Subject: [ceph-users] Re: replace osd with Octopus
Here is the context. https://docs.ceph.com/en/latest/mgr/orchestrator/#replace-an-osd
When disk is broken, 1) orch osd rm <svc_id(s)> --replace [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file>
Step #1 marks OSD "destroyed". I assume it has the same effect as "ceph osd destroy". And that keeps OSD "in", no PG remapping and cluster is in "degrade" state.
After step #3, OSD will be "up" and "in", data will be recovered back to new disk. Is that right?
Yes.
Is cluster "degrade" or "healthy" during such recovery?
It will be degraded, because there are fewer copies of some data available than during normal operation. Clients will continue to access all data.
For another option, the difference is no "--replace" in step #1. 1) orch osd rm <svc_id(s)> [--force] 2) Replace disk. 3) ceph orch apply osd -i <osd_spec_file>
Step #1 evacuates PGs from OSD and removes it from cluster. If disk is broken or OSD daemon is down, is this evacuation still going to work?
Yes, of course - broken drives are the typical reason for removing OSDs.
Is it going to take a while if there is lots data on this disk?
Yes, depending on what "a while" means to you, the size of the cluster, whether the pool is replicated or EC, and whether these are HDDs or SSDs.
After step #3, PGs will be rebalanced/remapped again when new OSD joins the cluster.
I think, to replace with the same disk model, option #1 is preferred, to replace with different disk model, it needs to be
Thanks! Tony option #2.
I haven't tried it under Octopus, but I don't think this is strictly true. If you replace it with a different model that is approximately the same size, everything will be fine. Through Luminous and I think Nautilus at least, if you `destroy` and replace with a larger drive, the CRUSH weight of the OSD will still reflect that of the old drive. You could then run `ceph osd crush reweight` after deploying to adjust the size. You could record the CRUSH weights of all your drive models for initial OSD deploys, or you could `ceph osd tree` and look for another OSD of the same model, and set the CRUSH weight accordingly.
If you replace with a smaller drive, your cluster will lose a small amount of usable capacity. If you replace with a larger drive, the cluster may or may not enjoy a slight increase in capacity - that depends on replication strategy, rack/host weights, etc.
My personal philosophy on drive replacements:
o Build OSDs with `-dmcrypt` so that you don't have to worry about data if/when you RMA or recycle bad drives. RMAs are a hassle, so pick a certain value threshold before a drive is worth the effort. This might be in the $250-500 range for example, which means that for many HDDs it isn't worth RMAing them.
o If you have an exact replacement, use it
o When buying spares, buy the largest size drive you have deployed - or will deploy within the next year or so. That way you know that your spares can take the place of any drive you have, so you don't have to maintain stock of more than one size. Worst case you don't immediately make good use of that extra capacity, but you may in the future as drives in other failure domains fail and are replaced. Be careful, though of mixing drives that a lot different in size. Mixing 12 and 14 TB drives, even 12 and 16 is usually no big deal, but if you mix say 1TB and 16 TB drives, you can end up exceeding `mon_max_pg_per_osd`. Which is one reason why I like to increase it from the default value to, say, 400. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Give my above understanding, all-to-all is no difference from one-to-all. In either case, PGs of one disk are remapped to others.
I must be missing something seriously:)
It’s a bit subtle, but I think part of what Frank is getting at is that when OSDs are backfilled / recovered sequentially, some data ends up being moved more than once. If one batches up such changes, data shouldn’t move an once. There are other factors too, like decreased capacity from letting failed drives pile up, and the impact of peering if one activates a large number of OSDs at the same time. Notably, one has to be careful when *removing* OSDs in batches to not cause PGs to go inactive. In Ceph there are often multiple ways to do a thing, with pros and cons. Sometimes there’s value in keeping it simple, especially for execution by 24/7 NOC personnel who have a lot going on.
participants (4)
-
Anthony D'Atri
-
Eugen Block
-
Frank Schilder
-
Tony Liu