How to repair the OSDs while WAL/DB device breaks down
hi, everyone, I have a question about repairing the broken WAL/DB device. I have a cluster with 8 OSDs, and 4 WAL/DB devices(1 OSD per WAL/DB device), and hwo can I repair the OSDs quickly if one WAL/DB device breaks down without rebuilding the them? Thanks.
hi, everyone, I have a question about repairing the broken WAL/DB device.
I have a cluster with 8 OSDs, and 4 WAL/DB devices(1 OSD per WAL/DB device), and hwo can I repair the OSDs quickly if
one WAL/DB device breaks down without rebuilding the them? Thanks.
I think this is one of the parameters one has to take into account when doing the risk assessment of using one fast device for many slow data devices when setting up OSDs. If/when the WAL/DB breaks, you should be prepared to rebuild the OSDs related to the broken WAL/DB device. From what I remember, the WAL+DB contains things that actually are vital for the OSD, so it's probably not a matter of running some magic command and recreate the information from the data device. For old filestore journals, one could somewhat live with a broken journaling device and recreate that, but I don't think that is true for bluestore DBs. -- May the most significant bit of your life be positive.
Janne, Thanks for your reply. To reduce the cost of recovering OSDs while WAL/DB device is down, maybe I have no choice but add more WAL/DB devices. On 2023/3/15 15:04, Janne Johansson wrote:
hi, everyone, I have a question about repairing the broken WAL/DB device.
I have a cluster with 8 OSDs, and 4 WAL/DB devices(1 OSD per WAL/DB device), and hwo can I repair the OSDs quickly if
one WAL/DB device breaks down without rebuilding the them? Thanks. I think this is one of the parameters one has to take into account when doing the risk assessment of using one fast device for many slow data devices when setting up OSDs. If/when the WAL/DB breaks, you should be prepared to rebuild the OSDs related to the broken WAL/DB device. From what I remember, the WAL+DB contains things that actually are vital for the OSD, so it's probably not a matter of running some magic command and recreate the information from the data device.
For old filestore journals, one could somewhat live with a broken journaling device and recreate that, but I don't think that is true for bluestore DBs.
Den tors 16 mars 2023 kl 06:42 skrev Norman <norman.kern@gmx.com>:
Janne,
Thanks for your reply. To reduce the cost of recovering OSDs while WAL/DB device is down, maybe I have no choice but add more WAL/DB devices.
We do run one ssd-or-nvme for several OSD hdd drives and have not seen this as a problem in itself. For us, hdds still break down more often than the flash drives, though we choose flash drives with DWPD > 3 to make sure it can sustain a lot of writes over time. If you use (please don't!) cheap consumer ssds or whatever, then I think they will break down as often as spin drives, and having less or one such per hdd would be a wiser choice. Still, we try to build our clusters so we can handle a whole box falling off, which means 8 or 12 hdd drives are gone at the same time, so compared to this, having a single SSD take out 3-4-5 OSDs is not a complete disaster and well within the expected margins. One has to take into account that boxes will crash, drives will fail, processes will crash sometimes and so on, this is why we build clusters, so that other units can help share the load when (not if) it happens. -- May the most significant bit of your life be positive.
Janne, Thanks for your advice. I'll have a try. :) On 2023/3/16 15:00, Janne Johansson wrote:
Janne,
Thanks for your reply. To reduce the cost of recovering OSDs while WAL/DB device is down, maybe I have no choice but add more WAL/DB devices. We do run one ssd-or-nvme for several OSD hdd drives and have not seen
Den tors 16 mars 2023 kl 06:42 skrev Norman <norman.kern@gmx.com>: this as a problem in itself. For us, hdds still break down more often than the flash drives, though we choose flash drives with DWPD > 3 to make sure it can sustain a lot of writes over time. If you use (please don't!) cheap consumer ssds or whatever, then I think they will break down as often as spin drives, and having less or one such per hdd would be a wiser choice.
Still, we try to build our clusters so we can handle a whole box falling off, which means 8 or 12 hdd drives are gone at the same time, so compared to this, having a single SSD take out 3-4-5 OSDs is not a complete disaster and well within the expected margins.
One has to take into account that boxes will crash, drives will fail, processes will crash sometimes and so on, this is why we build clusters, so that other units can help share the load when (not if) it happens.
participants (2)
-
Janne Johansson
-
Norman