25 Feb
2021
25 Feb
'21
10:50 p.m.
Hi all,
My cephfs MDS is reporting damaged metadata following the addition (and
remapping) of 12 new OSDs.
`ceph tell mds.database-0 damage ls` reports ~85 files damaged. All of type
"backtrace" which is very concerning.
` ceph tell mds.database-0 scrub start / recursive repair` seems to have no
effect on the damage. What does this sort of damage mean? Is there anything
I can do to recover these files?
ceph status reports:
cluster:
id: 692905c0-f271-4cd8-9e43-1c32ef8abd13
health: HEALTH_ERR
1 MDSs report damaged metadata
630 pgs not deep-scrubbed in time
630 pgs not scrubbed in time
services:
mon: 3 daemons, quorum database-0,file-server,webhost (age 37m)
mgr: webhost(active, since 3d), standbys: file-server, database-0
mds: cephfs:1 {0=database-0=up:active} 2 up:standby
osd: 48 osds: 48 up (since 56m), 48 in (since 13d); 10 remapped pgs
task status:
scrub status:
mds.database-0: idle
data:
pools: 7 pools, 633 pgs
objects: 60.82M objects, 231 TiB
usage: 336 TiB used, 246 TiB / 582 TiB avail
pgs: 623 active+clean
6 active+remapped+backfilling
4 active+remapped+backfill_wait
Thanks for the help.
Best,
Ricardo
25 Feb
25 Feb
11:06 p.m.
Hello Ricardo,
On Thu, Feb 25, 2021 at 11:51 AM <ricardo.re.azevedo(a)gmail.com> wrote:
Hi all,
My cephfs MDS is reporting damaged metadata following the addition (and
remapping) of 12 new OSDs.
`ceph tell mds.database-0 damage ls` reports ~85 files damaged. All of type
"backtrace" which is very concerning.
It is not concerning, actually. This just indicates that the reverse
link of the file's object data to its path in the file system is
incorrect.
` ceph tell mds.database-0 scrub start / recursive
repair` seems to have no
effect on the damage. What does this sort of damage mean? Is there anything
I can do to recover these files?
Scrubbing should correct it. Try "recursive repair force" to see if
that helps. "force" will cause the MDS to revisit metadata that has
been scrubbed previously but unchanged since then.
--
Patrick Donnelly, Ph.D.
He / Him / His
Principal Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
26 Feb
26 Feb
11:16 p.m.
Thanks for the advice and info regarding the error.
I tried ` ceph tell mds.database-0 scrub start / recursive repair force` and it didn't
help. Is there anything else I can try? Or manually fix the links?
Best,
Ricardo
-----Original Message-----
From: Patrick Donnelly <pdonnell(a)redhat.com>
Sent: Thursday, February 25, 2021 12:06 PM
To: ricardo.re.azevedo(a)gmail.com
Cc: ceph-users <ceph-users(a)ceph.io>
Subject: Re: [ceph-users] MDSs report damaged metadata
Hello Ricardo,
On Thu, Feb 25, 2021 at 11:51 AM <ricardo.re.azevedo(a)gmail.com> wrote:
Hi all,
My cephfs MDS is reporting damaged metadata following the addition
(and
remapping) of 12 new OSDs.
`ceph tell mds.database-0 damage ls` reports ~85 files damaged. All of
type "backtrace" which is very concerning.
It is not concerning, actually. This just indicates that the reverse link of the
file's object data to its path in the file system is incorrect.
` ceph tell mds.database-0 scrub start / recursive
repair` seems to
have no effect on the damage. What does this sort of damage mean? Is
there anything I can do to recover these files?
Scrubbing should correct it. Try "recursive repair force" to see if that helps.
"force" will cause the MDS to revisit metadata that has been scrubbed previously
but unchanged since then.
--
Patrick Donnelly, Ph.D.
He / Him / His
Principal Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
1157
days inactive
1158
days old
2 comments
2 participants
participants (2)
-
Patrick Donnelly -
ricardo.re.azevedo＠gmail.com