PG damaged "failed_repair" - ceph-users

26 Feb 2024

Hi,

We're a student club from Montréal where we host an Openstack cloud with 
a Ceph backend for storage of virtual machines and volumes using rbd.

Two weeks ago we received an email from our ceph cluster saying that 
some pages were damaged. We ran "sudo ceph pg repair <pg-id>" but then 
there was an I/O error on the disk during the recovery ("An 
unrecoverable disk media error occurred on Disk 4 in Backplane 1 of 
Integrated RAID Controller 1." and "Bad block medium error is detected 
at block 0x1377e2ad on Virtual Disk 3 on Integrated RAID Controller 1." 
messages on iDRAC).

After that, the PG we tried to repair was in the state 
"active+recovery_unfound+degraded". After a week, we ran the command 
"sudo ceph pg 2.1b mark_unfound_lost revert" to try to recover the 
damaged PG. We tried to boot the virtual machine that had crashed 
because of this incident, but the volume seemed to have been completely 
erased, the "mount" command said there was no filesystem on it, so we 
recreated the VM from a backup.

A few days later, the same PG was once again damaged, and since we knew 
the physical disk on the OSD hosting one part of the PG had problems, we 
tried to "out" the OSD from the cluster. That resulted in the two other 
OSDs hosting copies of the problematic PG to go down, which caused 
timeouts on our virtual machines, so we put the OSD back in.

We then tried to repair the PG again, but that failed and the PG is now 
"active+clean+inconsistent+failed_repair", and whenever it goes down, 
two other OSDs from two other hosts go down too after a few minutes, so 
it's impossible to replace the disk right now, even if we have new ones 
available.

We have backups for most of our services, but it would be very 
disrupting to delete the whole cluster, and we don't know that to do 
with the broken PG and the OSD that can't be shut down.

Any help would be really appreciated, we're not experts with Ceph and 
Openstack, and it's likely we handled things wrong at some point, but we 
really want to go back to a healthy Ceph.

Here are some information about our cluster :

romain:step@alpha-cen ~  $ sudo ceph health detail
HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent
[ERR] OSD_SCRUB_ERRORS: 1 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
     pg 2.1b is active+clean+inconsistent+failed_repair, acting [3,11,0]

romain:step@alpha-cen ~  $ sudo ceph osd tree
ID  CLASS  WEIGHT    TYPE NAME           STATUS  REWEIGHT  PRI-AFF
-1         70.94226  root default
-7         20.00792      host alpha-cen
  3    hdd   1.81879          osd.3           up   1.00000  1.00000
  6    hdd   1.81879          osd.6           up   1.00000  1.00000
12    hdd   1.81879          osd.12          up   1.00000  1.00000
13    hdd   1.81879          osd.13          up   1.00000  1.00000
15    hdd   1.81879          osd.15          up   1.00000  1.00000
16    hdd   9.09520          osd.16          up   1.00000  1.00000
17    hdd   1.81879          osd.17          up   1.00000  1.00000
-5         23.64874      host beta-cen
  1    hdd   5.45749          osd.1           up   1.00000  1.00000
  4    hdd   5.45749          osd.4           up   1.00000  1.00000
  8    hdd   5.45749          osd.8           up   1.00000  1.00000
11    hdd   5.45749          osd.11          up   1.00000  1.00000
14    hdd   1.81879          osd.14          up   1.00000  1.00000
-3         27.28560      host gamma-cen
  0    hdd   9.09520          osd.0           up   1.00000  1.00000
  5    hdd   9.09520          osd.5           up   1.00000  1.00000
  9    hdd   9.09520          osd.9           up   1.00000  1.00000

romain:step@alpha-cen ~  $ sudo rados list-inconsistent-obj 2.1b
{"epoch":9787,"inconsistents":[]}

romain:step@alpha-cen ~  $ sudo ceph pg 2.1b query

https://pastebin.com/gsKCPCjr

Best regards,

Romain Lebbadi-Breteau