Natuilus: Taking out OSDs that are 'Failure Pending'
Hello. It's been a while. I have a Nautilus cluster with 72 x 12GB HDD OSDs (BlueStore) and mostly of EC 8+2 pools/PGs. It's been working great - some nodes went nearly 900 days without a reboot. As of yesterday I found that I have 3 OSDs with a Smart status of 'Pending Failure'. New drives are ordered and will be here next week. There is a procedure in the documentation for replacing an OSD, but I can't do that directly until I receive the drives. My inclination is to mark these 3 OSDs 'OUT' before they crash completely, but I want to confirm my understanding of Ceph's response to this. Mainly, given my EC pools (or replicated pools for that matter), if I mark all 3 OSD out all at once will I risk data loss? If I have it right, marking an OSD out will simply cause Ceph to move all of the PG shards from that OSD to other OSDs, so no major risk of data loss. However, if it would be better to do them one per day or something, I'd rather be safe. I also assume that I should wait for the rebalance to complete before I initiate the replacement procedure. Your thoughts? Thanks. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu
Marking them OUT first is the way to go. As long as the osds stay UP, they can and will participate in the recovery. How many you can mark out at one time will depend on how sensitive your client i/o is to background recovery, and all of the related tunings. If you have the hours/days to spare, it is definitely easier on the cluster to do them one at a time. Thank you, Josh Beaman From: Dave Hall <kdhall@binghamton.edu> Date: Friday, August 4, 2023 at 8:45 AM To: ceph-users <ceph-users@ceph.io> Cc: anthony.datri <anthony.datri@gmail.com> Subject: [EXTERNAL] [ceph-users] Natuilus: Taking out OSDs that are 'Failure Pending' Hello. It's been a while. I have a Nautilus cluster with 72 x 12GB HDD OSDs (BlueStore) and mostly of EC 8+2 pools/PGs. It's been working great - some nodes went nearly 900 days without a reboot. As of yesterday I found that I have 3 OSDs with a Smart status of 'Pending Failure'. New drives are ordered and will be here next week. There is a procedure in the documentation for replacing an OSD, but I can't do that directly until I receive the drives. My inclination is to mark these 3 OSDs 'OUT' before they crash completely, but I want to confirm my understanding of Ceph's response to this. Mainly, given my EC pools (or replicated pools for that matter), if I mark all 3 OSD out all at once will I risk data loss? If I have it right, marking an OSD out will simply cause Ceph to move all of the PG shards from that OSD to other OSDs, so no major risk of data loss. However, if it would be better to do them one per day or something, I'd rather be safe. I also assume that I should wait for the rebalance to complete before I initiate the replacement procedure. Your thoughts? Thanks. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Fri, Aug 04, 2023 at 09:44:57AM -0400, Dave Hall wrote:
My inclination is to mark these 3 OSDs 'OUT' before they crash completely, but I want to confirm my understanding of Ceph's response to this. Mainly, given my EC pools (or replicated pools for that matter), if I mark all 3 OSD out all at once will I risk data loss?
It depends on your crush map and failure domain layout. In the unlikeliest and unluckiest case, all those 3 OSDs are in different failure domains, and some data has 1 replica on each of those OSDs. In that situation, if you take them out simultaneously, you would lose data. If you're unsure, then do them one at a time and wait for the rebalance/backfill to complete before doing the next. We arrange our OSDs so that the failure domain is the rack; losing an entire rack is safe (and we've had that happen) so we know it's safe to pull any number of OSDs in the same rack and we won't lose data. Dave -- ** Dave Holland ** Systems Support -- Informatics Systems Group ** ** dh3@sanger.ac.uk ** Wellcome Sanger Institute, Hinxton, UK ** -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.
Dave, Actually, my failure domain is OSD since I so far only have 9 OSD nodes but EC 8+2. However, the drives are still functioning, except that one has failed multiple times in the last few days, requiring a node power-cycle to recover. I will certainly mark that one out immediately. The other two pending failures are behaving more politely, so I am assuming that the cluster could copy the data elsewhere as part of the rebalance. I think I'm also concerned about the rebalance process moving data to these drives with pending failures. Since I'm EC 8+2, perhaps it is safe to mark two out simultaneously? Thanks. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu On Fri, Aug 4, 2023 at 10:16 AM Dave Holland <dh3@sanger.ac.uk> wrote:
On Fri, Aug 04, 2023 at 09:44:57AM -0400, Dave Hall wrote:
My inclination is to mark these 3 OSDs 'OUT' before they crash completely, but I want to confirm my understanding of Ceph's response to this. Mainly, given my EC pools (or replicated pools for that matter), if I mark all 3 OSD out all at once will I risk data loss?
It depends on your crush map and failure domain layout. In the unlikeliest and unluckiest case, all those 3 OSDs are in different failure domains, and some data has 1 replica on each of those OSDs. In that situation, if you take them out simultaneously, you would lose data. If you're unsure, then do them one at a time and wait for the rebalance/backfill to complete before doing the next.
We arrange our OSDs so that the failure domain is the rack; losing an entire rack is safe (and we've had that happen) so we know it's safe to pull any number of OSDs in the same rack and we won't lose data.
Dave -- ** Dave Holland ** Systems Support -- Informatics Systems Group ** ** dh3@sanger.ac.uk ** Wellcome Sanger Institute, Hinxton, UK **
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.
On Fri, Aug 4, 2023 at 11:33 AM Dave Hall <kdhall@binghamton.edu> wrote:
Dave,
Actually, my failure domain is OSD since I so far only have 9 OSD nodes but EC 8+2. However, the drives are still functioning, except that one has failed multiple times in the last few days, requiring a node power-cycle to recover. I will certainly mark that one out immediately.
The other two pending failures are behaving more politely, so I am assuming that the cluster could copy the data elsewhere as part of the rebalance. I think I'm also concerned about the rebalance process moving data to these drives with pending failures.
Since I'm EC 8+2, perhaps it is safe to mark two out simultaneously?
Dave, You should be able to mark out two OSDs simultaneously without worry as long as you have enough space, etc. When you mark an OSD out, it still participates in the cluster as long as the OSD remains up and is able to aid in the backfilling process. Thus, you'll also want to avoid stopping/downing the OSDs until backfilling completes. Following that logic: if you stop both OSDs before backfilling completes, you will put yourself in a bad spot. If all PGs are active+clean, you may both a) out the two OSDs and b) stop/down *only the one* imminently failing OSD (leaving the second OSD being drained still up) and things should also be fine... but you will be vulnerable to blocked ops/unavailable data if _subsequent_ OSDs fail unexpectedly, including the second OSD being out'd, depending upon your CRUSH map and cluster status. Note that if your intent is to purge the OSD after it is drained, I believe you should do a `ceph osd crush reweight osd.X 0` and not an `ceph out osd.X` or `ceph osd reweight osd.X 0` as it should result in slightly less net data movement. Cheers, Tyler
participants (4)
-
Beaman, Joshua
-
Dave Hall
-
Dave Holland
-
Tyler Stachecki