MDSs report damaged metadata
Hi all, My cephfs MDS is reporting damaged metadata following the addition (and remapping) of 12 new OSDs. `ceph tell mds.database-0 damage ls` reports ~85 files damaged. All of type "backtrace" which is very concerning. ` ceph tell mds.database-0 scrub start / recursive repair` seems to have no effect on the damage. What does this sort of damage mean? Is there anything I can do to recover these files?
ceph status reports: cluster:
id: 692905c0-f271-4cd8-9e43-1c32ef8abd13 health: HEALTH_ERR 1 MDSs report damaged metadata 630 pgs not deep-scrubbed in time 630 pgs not scrubbed in time services: mon: 3 daemons, quorum database-0,file-server,webhost (age 37m) mgr: webhost(active, since 3d), standbys: file-server, database-0 mds: cephfs:1 {0=database-0=up:active} 2 up:standby osd: 48 osds: 48 up (since 56m), 48 in (since 13d); 10 remapped pgs task status: scrub status: mds.database-0: idle data: pools: 7 pools, 633 pgs objects: 60.82M objects, 231 TiB usage: 336 TiB used, 246 TiB / 582 TiB avail pgs: 623 active+clean 6 active+remapped+backfilling 4 active+remapped+backfill_wait Thanks for the help. Best, Ricardo
Hello Ricardo, On Thu, Feb 25, 2021 at 11:51 AM <ricardo.re.azevedo@gmail.com> wrote:
Hi all,
My cephfs MDS is reporting damaged metadata following the addition (and remapping) of 12 new OSDs. `ceph tell mds.database-0 damage ls` reports ~85 files damaged. All of type "backtrace" which is very concerning.
It is not concerning, actually. This just indicates that the reverse link of the file's object data to its path in the file system is incorrect.
` ceph tell mds.database-0 scrub start / recursive repair` seems to have no effect on the damage. What does this sort of damage mean? Is there anything I can do to recover these files?
Scrubbing should correct it. Try "recursive repair force" to see if that helps. "force" will cause the MDS to revisit metadata that has been scrubbed previously but unchanged since then. -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Thanks for the advice and info regarding the error. I tried ` ceph tell mds.database-0 scrub start / recursive repair force` and it didn't help. Is there anything else I can try? Or manually fix the links? Best, Ricardo -----Original Message----- From: Patrick Donnelly <pdonnell@redhat.com> Sent: Thursday, February 25, 2021 12:06 PM To: ricardo.re.azevedo@gmail.com Cc: ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] MDSs report damaged metadata Hello Ricardo, On Thu, Feb 25, 2021 at 11:51 AM <ricardo.re.azevedo@gmail.com> wrote:
Hi all,
My cephfs MDS is reporting damaged metadata following the addition (and remapping) of 12 new OSDs. `ceph tell mds.database-0 damage ls` reports ~85 files damaged. All of type "backtrace" which is very concerning.
It is not concerning, actually. This just indicates that the reverse link of the file's object data to its path in the file system is incorrect.
` ceph tell mds.database-0 scrub start / recursive repair` seems to have no effect on the damage. What does this sort of damage mean? Is there anything I can do to recover these files?
Scrubbing should correct it. Try "recursive repair force" to see if that helps. "force" will cause the MDS to revisit metadata that has been scrubbed previously but unchanged since then. -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
participants (2)
-
Patrick Donnelly
-
ricardo.re.azevedo＠gmail.com