4 Jun
2020
4 Jun
'20
1:09 p.m.
Hi,
It turns out I was mapping to a problematic OSD. In this case OSD 3313.
After disabling the OSD with systemctl on the host, recovery has picked up
again and mapped the pgs to new osds.
For prosperity, I ran smartctl on osd.3313's device and then I noticed:
5 Reallocated_Sector_Ct 0x0033 092 092 010 Pre-fail Always
- *31688*
Lots of reallocated sectors, so the drive was "working" but not usable.
In the end it had nothing to do with Ceph at all.
Regards,
On Thu, Jun 4, 2020 at 1:59 PM Thomas Bennett <thomas(a)ska.ac.za> wrote:
Hi,
I have 15628 misplaced objects that are currently backfilling as follows:
1. pgid:14.3ce1 from:osd.1321 to:osd.3313
2. pgid:14.4dd9 from:osd.1693 to:osd.2980
3. pgid:14.680b from:osd.362 to:osd.3313
These are remnant backfills from a pg-upmap/rebalance campaign after we've
added 2 new racks worth of osds to our cluster.
Our mon db is bloated so I'm wanting to trim the mon db before continuing
the next pg-upmap/rebalance campaign.
So, my question is:
Is there any way I can speed up the backfill process on these individual
osds?
Or hints to trace out why these are so slow?
Regards
--
Thomas Bennett
Storage Engineer at SARAO