The majority of the pools have ‘replicated size 3 min_size 2’.
Groovy.
I do see a few pools such as .rgw.control and a few others have ‘replicated size 3 min_size 1’.
Not a good way to run. Set min_size to 2 after you get healthy.
I am not using erasure encoding and none of the pools are set to ‘replicated size 3 min_size 3’.
Odd that you’re in this situation. You might increase the retries in your crush rules. You might also set min_ size temporarily to 1 on pool #0, which may let these PGs activate and recover, then immediately set back to 2, then investigate if all PGs now have a full acting set. NB There is some risk here.
Thank you,
Shain
From: Anthony D'Atri <anthony.datri@gmail.com> Date: Sunday, October 13, 2024 at 11:29 AM To: Shain Miley <SMiley@npr.org> Cc: ceph-users@ceph.io <ceph-users@ceph.io> Subject: Re: [ceph-users] Reduced data availability: 3 pgs inactive, 3 pgs down !-------------------------------------------------------------------| External Email - Use Caution
|-------------------------------------------------------------------!
When you get the cluster healthy, redeploy those Filestore OSDs as BlueStore. Not before.
Does you r pool have size=3, min_size=3? Is this a replicated pool? Or EC 2,1?
Don’t mark lost, there are things we can do. I don’t want to suggest anything until you share the above info.
On Oct 13, 2024, at 10:00 AM, Shain Miley <SMiley@npr.org> wrote:
Hello,
I am seeing the following information after reviewing ‘ceph health detail’:
[WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive, 3 pgs down
pg 0.1a is down, acting [234,35]
pg 0.20 is down, acting [226,267]
pg 0.2f is down, acting [227,161]
When I query each of those pgs I see the following message on each of them:
"peering_blocked_by": [
{
"osd": 233,
"current_lost_at": 0,
"comment": "starting or marking this osd lost may let us proceed"
}
Osd.233 crashed a while ago and when I try to start it the log shows some sort of issue with the filesystem:
ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)
1: (()+0x12980) [0x7f2779617980]
2: (gsignal()+0xc7) [0x7f27782c9fb7]
3: (abort()+0x141) [0x7f27782cb921]
4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b2) [0x556ebe773ddf]
5: (FileStore::_do_transaction(ceph::os::Transaction&, unsigned long, int, ThreadPool::TPHandle*, char const*)+0x62b3) [0x556ebebe2753]
6: (FileStore::_do_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, unsigned long, ThreadPool::TPHandle*, char const*)+0x48) [0x556ebebe3f38]
7: (JournalingObjectStore::journal_replay(unsigned long)+0x105a) [0x556ebebfc56a]
8: (FileStore::mount()+0x438a) [0x556ebebda82a]
9: (OSD::init()+0x4d1) [0x556ebe80fdc1]
10: (main()+0x3f8c) [0x556ebe77ad2c]
11: (__libc_start_main()+0xe7) [0x7f27782acbf7]
12: (_start()+0x2a) [0x556ebe78fc4a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
At this point I am thinking about either running an xfs repair on osd.233 and trying to see if I can get it back up (once the pgs are healthy again I would likey zap/readd or replace the drive).
Another option it sounds like is to mark the osd as lost.
I am just looking for advice on what exactly I should do next to try to minimize the chances of any data loss.
Here is the query output for each of those pgs: https://urldefense.com/v3/__https://pastebin.com/YbfnpZGC__;!!Iwwt!XTVUuKiQDmZ8ZXQP-pHoxFFWYAIntSqVBuXcigFVVWbYMtpJTcQeg4BzgQQxWSAhs1BKujMNHx4rDIhAStU$<https://urldefense.com/v3/__https:/pastebin.com/YbfnpZGC__;!!Iwwt!XTVUuKiQDmZ8ZXQP-pHoxFFWYAIntSqVBuXcigFVVWbYMtpJTcQeg4BzgQQxWSAhs1BKujMNHx4rDIhAStU$>
Thank you,
Shain
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io