21 May
2021
21 May
'21
6:10 p.m.
Hi all
An accidental power failure happened.
That resulted CephFS offline and cannot be mounted.
I have 3 MDS daemons but it complains "1 mds daemon damaged".
It seems a PG of cephfs_metadata is inconsistent. I tried to repair, but doesn't get
it repaired.
How do I repair the damaged MDS and bring the CephFS up/online?
Details are included below.
Many thanks in advance.
Sagara
# ceph -s
cluster:
id: abc...
health: HEALTH_ERR
1 filesystem is degraded
1 filesystem is offline
1 mds daemon damaged
4 scrub errors
Possible data damage: 1 pg inconsistent
services:
mon: 3 daemons, quorum a,b,c (age 107s)
mgr: a(active, since 22m), standbys: b, c
mds: cephfs:0/1 3 up:standby, 1 damaged
osd: 3 osds: 3 up (since 96s), 3 in (since 96s)
data:
pools: 3 pools, 192 pgs
objects: 281.05k objects, 327 GiB
usage: 2.4 TiB used, 8.1 TiB / 11 TiB avail
pgs: 191 active+clean
1 active+clean+inconsistent
# ceph health detail
HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged; 4
scrub errors; Possible data damage: 1 pg inconsistent
FS_DEGRADED 1 filesystem is degraded
fs cephfs is degraded
MDS_ALL_DOWN 1 filesystem is offline
fs cephfs is offline because no MDS is active for it.
MDS_DAMAGE 1 mds daemon damaged
fs cephfs mds.0 is damaged
OSD_SCRUB_ERRORS 4 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
pg 2.44 is active+clean+inconsistent, acting [0,2,1]
# ceph osd lspools
2 cephfs_metadata
3 cephfs_data
4 rbd
# ceph pg repair 2.44
# ceph -w
2021-05-22 01:48:04.775783 osd.0 [ERR] 2.44 shard 0 soid 2:22efaf6a:::200.00006048:head :
candidate size 1540096 info size 1555896 mismatch
2021-05-22 01:48:04.775786 osd.0 [ERR] 2.44 shard 1 soid 2:22efaf6a:::200.00006048:head :
candidate size 1540096 info size 1555896 mismatch
2021-05-22 01:48:04.775787 osd.0 [ERR] 2.44 shard 2 soid 2:22efaf6a:::200.00006048:head :
candidate size 1441792 info size 1555896 mismatch
2021-05-22 01:48:04.775789 osd.0 [ERR] 2.44 soid 2:22efaf6a:::200.00006048:head : failed
to pick suitable object info
2021-05-22 01:48:04.775849 osd.0 [ERR] repair 2.44 2:22efaf6a:::200.00006048:head : on
disk size (1540096) does not match object info size (1555896) adjusted for ondisk to
(1555896)
2021-05-22 01:48:04.787167 osd.0 [ERR] 2.44 repair 4 errors, 0 fixed
--- End of detail ---
21 May
21 May
7:09 p.m.
Hi,
I went through similar trouble just this week [1], but the root cause
seems different so it probably won't apply to your case.
Which version of ceph are you running? There are a couple of reports
with similar error messages, e. g. [2], it may already been resolved.
Can you share
rados list-inconsistent-obj 2.44
and
ceph tell mds.<MDS> damage ls
The pool size is 3, right?
Regards,
Eugen
Zitat von Sagara Wijetunga <sagarawmw(a)yahoo.com>om>:
Hi all
An accidental power failure happened.
That resulted CephFS offline and cannot be mounted.
I have 3 MDS daemons but it complains "1 mds daemon damaged".
It seems a PG of cephfs_metadata is inconsistent. I tried to repair,
but doesn't get it repaired.
How do I repair the damaged MDS and bring the CephFS up/online?
Details are included below.
Many thanks in advance.
Sagara
# ceph -s
cluster:
id: abc...
health: HEALTH_ERR
1 filesystem is degraded
1 filesystem is offline
1 mds daemon damaged
4 scrub errors
Possible data damage: 1 pg inconsistent
services:
mon: 3 daemons, quorum a,b,c (age 107s)
mgr: a(active, since 22m), standbys: b, c
mds: cephfs:0/1 3 up:standby, 1 damaged
osd: 3 osds: 3 up (since 96s), 3 in (since 96s)
data:
pools: 3 pools, 192 pgs
objects: 281.05k objects, 327 GiB
usage: 2.4 TiB used, 8.1 TiB / 11 TiB avail
pgs: 191 active+clean
1 active+clean+inconsistent
# ceph health detail
HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds
daemon damaged; 4 scrub errors; Possible data damage: 1 pg
inconsistent
FS_DEGRADED 1 filesystem is degraded
fs cephfs is degraded
MDS_ALL_DOWN 1 filesystem is offline
fs cephfs is offline because no MDS is active for it.
MDS_DAMAGE 1 mds daemon damaged
fs cephfs mds.0 is damaged
OSD_SCRUB_ERRORS 4 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
pg 2.44 is active+clean+inconsistent, acting [0,2,1]
# ceph osd lspools
2 cephfs_metadata
3 cephfs_data
4 rbd
# ceph pg repair 2.44
# ceph -w
2021-05-22 01:48:04.775783 osd.0 [ERR] 2.44 shard 0 soid
2:22efaf6a:::200.00006048:head : candidate size 1540096 info size
1555896 mismatch
2021-05-22 01:48:04.775786 osd.0 [ERR] 2.44 shard 1 soid
2:22efaf6a:::200.00006048:head : candidate size 1540096 info size
1555896 mismatch
2021-05-22 01:48:04.775787 osd.0 [ERR] 2.44 shard 2 soid
2:22efaf6a:::200.00006048:head : candidate size 1441792 info size
1555896 mismatch
2021-05-22 01:48:04.775789 osd.0 [ERR] 2.44 soid
2:22efaf6a:::200.00006048:head : failed to pick suitable object info
2021-05-22 01:48:04.775849 osd.0 [ERR] repair 2.44
2:22efaf6a:::200.00006048:head : on disk size (1540096) does not
match object info size (1555896) adjusted for ondisk to (1555896)
2021-05-22 01:48:04.787167 osd.0 [ERR] 2.44 repair 4 errors, 0 fixed
--- End of detail ---
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:10 p.m.
Sorry, forgot the links:
[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/4WJ5IUANKYO…
[2] https://tracker.ceph.com/issues/43068
Zitat von Eugen Block <eblock(a)nde.ag>ag>:
Hi,
I went through similar trouble just this week [1], but the root
cause seems different so it probably won't apply to your case.
Which version of ceph are you running? There are a couple of reports
with similar error messages, e. g. [2], it may already been resolved.
Can you share
rados list-inconsistent-obj 2.44
and
ceph tell mds.<MDS> damage ls
The pool size is 3, right?
Regards,
Eugen
Zitat von Sagara Wijetunga <sagarawmw(a)yahoo.com>om>:
> Hi all
> An accidental power failure happened.
> That resulted CephFS offline and cannot be mounted.
> I have 3 MDS daemons but it complains "1 mds daemon damaged".
>
> It seems a PG of cephfs_metadata is inconsistent. I tried to
> repair, but doesn't get it repaired.
> How do I repair the damaged MDS and bring the CephFS up/online?
> Details are included below.
>
> Many thanks in advance.
> Sagara
>
>
>
>
> # ceph -s
>
> cluster:
>
> id: abc...
>
> health: HEALTH_ERR
>
> 1 filesystem is degraded
>
> 1 filesystem is offline
>
> 1 mds daemon damaged
>
> 4 scrub errors
>
> Possible data damage: 1 pg inconsistent
>
>
>
> services:
>
> mon: 3 daemons, quorum a,b,c (age 107s)
>
> mgr: a(active, since 22m), standbys: b, c
>
> mds: cephfs:0/1 3 up:standby, 1 damaged
>
> osd: 3 osds: 3 up (since 96s), 3 in (since 96s)
>
>
>
> data:
>
> pools: 3 pools, 192 pgs
>
> objects: 281.05k objects, 327 GiB
>
> usage: 2.4 TiB used, 8.1 TiB / 11 TiB avail
>
> pgs: 191 active+clean
>
> 1 active+clean+inconsistent
>
>
>
> # ceph health detail
>
> HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds
> daemon damaged; 4 scrub errors; Possible data damage: 1 pg
> inconsistent
>
> FS_DEGRADED 1 filesystem is degraded
>
> fs cephfs is degraded
>
> MDS_ALL_DOWN 1 filesystem is offline
>
> fs cephfs is offline because no MDS is active for it.
>
> MDS_DAMAGE 1 mds daemon damaged
>
> fs cephfs mds.0 is damaged
>
> OSD_SCRUB_ERRORS 4 scrub errors
>
> PG_DAMAGED Possible data damage: 1 pg inconsistent
>
> pg 2.44 is active+clean+inconsistent, acting [0,2,1]
>
>
>
> # ceph osd lspools
>
> 2 cephfs_metadata
>
> 3 cephfs_data
>
> 4 rbd
>
>
>
>
> # ceph pg repair 2.44
>
>
> # ceph -w
>
> 2021-05-22 01:48:04.775783 osd.0 [ERR] 2.44 shard 0 soid
> 2:22efaf6a:::200.00006048:head : candidate size 1540096 info size
> 1555896 mismatch
>
>
> 2021-05-22 01:48:04.775786 osd.0 [ERR] 2.44 shard 1 soid
> 2:22efaf6a:::200.00006048:head : candidate size 1540096 info size
> 1555896 mismatch
>
>
> 2021-05-22 01:48:04.775787 osd.0 [ERR] 2.44 shard 2 soid
> 2:22efaf6a:::200.00006048:head : candidate size 1441792 info size
> 1555896 mismatch
>
>
> 2021-05-22 01:48:04.775789 osd.0 [ERR] 2.44 soid
> 2:22efaf6a:::200.00006048:head : failed to pick suitable object info
>
> 2021-05-22 01:48:04.775849 osd.0 [ERR] repair 2.44
> 2:22efaf6a:::200.00006048:head : on disk size (1540096) does not
> match object info size (1555896) adjusted for ondisk to (1555896)
>
> 2021-05-22 01:48:04.787167 osd.0 [ERR] 2.44 repair 4 errors, 0 fixed
>
> --- End of detail ---
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
22 May
22 May
5:43 a.m.
Hi Eugen
Thanks for the reply.
Ceph Version:# ceph versionceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf)
nautilus (stable)
Can you share
rados list-inconsistent-obj 2.44
# rados list-inconsistent-obj 2.44
{"epoch":6996,"inconsistents":[]}
ceph tell mds.<MDS> damage ls
# ceph tell mds.a damage ls
2021-05-22 13:23:34.135 80bf25c00 0 client.4344312 ms_handle_reset on
v2:192.168.1.130:6810/3532878906
2021-05-22 13:23:34.146 80dcc2500 0 client.4344318 ms_handle_reset on
v2:192.168.1.130:6810/3532878906
Error EINVAL: MDS not active
The pool size is 3, right?
Yes, pool size is 3.
Above Ceph error messages shows "fs cephfs mds.0 is damaged". My MDS are named
a,b and c. Does mds.0 means mds.a?
Since I have 2 MDS not damaged, how do I recover and mount the CephFS?
Btw, our home directories are on CephFS, at the moment, no one can log in.
Sagara
7:08 a.m.
What does the MDS report in its logs from when it went down?
What size do you get when you run
rados -p cephfs_metadata stat 200.00006048
There's a similar report [3] suggesting to try to force an update on
the object info, you could give that a shot:
1. rados -p [cephfs_metadata] setomapval 200.00006048
temporary-key anything
2. ceph pg deep-scrub 2.44
3. Wait for the scrub to finish
4. rados -p [cephfs_metadata] rmomapkey 200.00006048 temporary-key
Above Ceph error messages shows "fs cephfs mds.0
is damaged". My MDS
are named a,b and c. Does mds.0 means mds.a?
Here mds.0 means the rank 0, I would assume that you only have one
rank, correct (one filesystem with standby MDS daemons)?
[3] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-May/034580.html
Zitat von Sagara Wijetunga <sagarawmw(a)yahoo.com>om>:
Hi Eugen
Thanks for the reply.
Ceph Version:# ceph versionceph version 14.2.11
(f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
Can you share
rados list-inconsistent-obj 2.44
# rados list-inconsistent-obj 2.44
{"epoch":6996,"inconsistents":[]}
ceph tell mds.<MDS> damage ls
# ceph tell mds.a damage ls
2021-05-22 13:23:34.135 80bf25c00 0 client.4344312 ms_handle_reset
on v2:192.168.1.130:6810/3532878906
2021-05-22 13:23:34.146 80dcc2500 0 client.4344318 ms_handle_reset
on v2:192.168.1.130:6810/3532878906
Error EINVAL: MDS not active
The pool size is 3, right?
Yes, pool size is 3.
Above Ceph error messages shows "fs cephfs mds.0 is damaged". My MDS
are named a,b and c. Does mds.0 means mds.a?
Since I have 2 MDS not damaged, how do I recover and mount the CephFS?
Btw, our home directories are on CephFS, at the moment, no one can log in.
Sagara
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:43 a.m.
On Saturday, May 22, 2021, 03:14:13 PM GMT+8, Eugen Block <eblock(a)nde.ag> wrote:
What does the MDS report in its logs from when it went down?
What size do you get when you run
rados -p cephfs_metadata stat 200.00006048
# rados -p cephfs_metadata stat 200.00006048cephfs_metadata/200.00006048 mtime 2021-05-20
22:47:30.000000, size 1555896
There's a similar report [3] suggesting to try to force an update on
the object info, you could give that a shot:
1. rados -p [cephfs_metadata] setomapval 200.00006048
temporary-key anything
2. ceph pg deep-scrub 2.44
3. Wait for the scrub to finish
4. rados -p [cephfs_metadata] rmomapkey 200.00006048 temporary-key
I gave it a try. Here are the detail:
Before try it:rados list-inconsistent-obj 2.44 --format=json-pretty{ "epoch":
6996, "inconsistents": []}
After trying the step 1:# rados list-inconsistent-obj 2.44 --format=json-pretty{
"epoch": 6996, "inconsistents": [ {
"object": { "name": "200.00006048",
"nspace": "", "locator": "",
"snap": "head", "version": 0
}, "errors": [], "union_shard_errors":
[ "obj_size_info_mismatch" ],
"shards": [::
That is, It created the info for PG 2.44.
It shows size differences as follows:shards": [ {
"osd": 0, "primary": true,
"errors": [ "obj_size_info_mismatch"
], "size": 1540096,
"object_info": { "size":
1555896,
"osd": 1, "primary": false,
"errors": [ "obj_size_info_mismatch"
], "size": 1540096,
"object_info": { "size": 1555896,
"osd": 2, "primary": false,
"errors": [
"obj_size_info_mismatch" ],
"size": 1441792, "object_info": {
"size": 1555896,
Please note, the physical file size of 200.00006048 on OSD 0 and 1 is 1540096.
Physical file size on OSD 2 is 1441792.
I understand file sizes of the 200.00006048 should be same on all OSDs.
What should I do in that case?
Please also note, "ceph pg deep-scrub 2.44" did not fix the PG 2.44.
Sagara
10:58 a.m.
Here are the physical file sizes of the "200.00006048*":
OSD.0:-rw-r--r-- 1 ceph ceph 1540096 May 20 22:47
/var/lib/ceph/osd/ceph-0/current/2.44_head/200.00006048__head_56F5F744__
OSD.1:-rw-r--r-- 1 ceph ceph 1540096 May 20 22:47
/var/lib/ceph/osd/ceph-1/current/2.44_head/DIR_4/DIR_4/200.00006048__head_56F5F744__2
OSD2:-rw-r--r-- 1 ceph ceph 1441792 May 20 22:47
/var/lib/ceph/osd/ceph-2/current/2.44_head/DIR_4/DIR_4/200.00006048__head_56F5F744__2
Sagara
11:45 a.m.
On Saturday, May 22, 2021, 03:14:13 PM GMT+8, Eugen Block <eblock(a)nde.ag> wrote:
What does the MDS report in its logs from when it went down?
NOTE: Power failure happened somewhere 2021-05-20 23:56:
Here are log messages from MDS.0 log:
2021-05-20 17:26:19.358 2192d80 1 mds.a Updating MDS map to version 8746 from mon.0
2021-05-20 23:56:43.129 1364480 0 set uid:gid to 167:167 (ceph:ceph)
2021-05-20 23:56:43.129 1364480 0 ceph version 14.2.11
(f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable), process ceph-mds, pid 1624
2021-05-20 23:56:43.255 1fd6d80 1 mds.a Updating MDS map to version 8747 from mon.0
2021-05-20 23:56:47.327 1fd6d80 1 mds.a Updating MDS map to version 8748 from mon.0
2021-05-20 23:56:47.327 1fd6d80 1 mds.a Monitors have assigned me to become a standby.
2021-05-20 23:56:47.344 1fd6d80 1 mds.a Updating MDS map to version 8749 from mon.0
2021-05-20 23:56:47.658 1fd6d80 1 mds.0.8749 handle_mds_map i am now mds.0.8749
2021-05-20 23:56:47.689 1fd6d80 1 mds.0.8749 handle_mds_map state change up:boot -->
up:replay
2021-05-20 23:56:47.689 1fd6d80 1 mds.0.8749 replay_start
2021-05-20 23:56:47.689 1fd6d80 1 mds.0.8749 recovery set is
2021-05-20 23:56:47.689 1fd6d80 1 mds.0.8749 waiting for osdmap 6958 (which blacklists
prior instance)
2021-05-20 23:56:48.165 2228880 0 mds.0.cache creating system inode with ino:0x100
2021-05-20 23:56:48.177 2228880 0 mds.0.cache creating system inode with ino:0x1
2021-05-20 23:56:52.223 2227200 0 mds.0.journaler.mdlog(ro) _finish_read got less than
expected (1555896)
2021-05-20 23:56:52.223 2229180 0 mds.0.log _replay journaler got error -22, aborting
2021-05-20 23:56:52.223 2229180 -1 log_channel(cluster) log [ERR] : Error loading MDS rank
0: (22) Invalid argument
2021-05-20 23:56:52.224 2229180 1 mds.a respawn!
--- begin dump of recent events ---
Sagara
12:29 p.m.
Out of the info now emerged so far seems Ceph client wanted to write an object of
size 1555896 but managed to write only 1555896 bytes to the journal.
I think what we need to do now is:1. Get the MDS.0 recover, discard if necessary part of
the object 200.00006048 and bring the MSD.0 up.
2. Do the same recovery for the MSD.1 as in step 1 and bring MDS.1 also up.
3. Above two steps to the most probability may bring CephFS up.
4. Once the CephFS is up, scan for corrupted files, remove them and bring from backup.
5. Get the MDS.2 to sync to MSD.0 or 1 and bring the cluster to sync'ed stage.
My question is, what exactly necessary to carry above step 1?
Sagara
12:34 p.m.
Sorry, above post has to be corrected as:"Out of the info now emerged so far seems
Ceph client wanted to write an object of size 1555896 but managed to write only 1540096
bytes to the journal."
Sagara
On Saturday, May 22, 2021, 08:29:34 PM GMT+8, Sagara Wijetunga
<sagarawmw(a)yahoo.com> wrote:
Out of the info now emerged so far seems Ceph client wanted to write an object of
size 1555896 but managed to write only 1555896 bytes to the journal.
I think what we need to do now is:1. Get the MDS.0 recover, discard if necessary part of
the object 200.00006048 and bring the MSD.0 up.
2. Do the same recovery for the MSD.1 as in step 1 and bring MDS.1 also up.
3. Above two steps to the most probability may bring CephFS up.
4. Once the CephFS is up, scan for corrupted files, remove them and bring from backup.
5. Get the MDS.2 to sync to MSD.0 or 1 and bring the cluster to sync'ed stage.
My question is, what exactly necessary to carry above step 1?
Sagara
1:19 p.m.
Sorry, above post has to be corrected as:"Out of
the info now
emerged so far seems Ceph client wanted to write an object of size
1555896 but managed to write only 1540096 bytes to the journal."
Yes, I would think so, too.
I think what we need to do now is:
1. Get the MDS.0 recover, discard if necessary part of the object
200.00006048 and bring the MSD.0 up.
Yes, I agree, I just can't tell what the best way is here, maybe
remove all three objects from the disks (make a backup before doing
that, just in case) and try the steps to recover the journal (also
make a backup of the journal first):
mds01:~ # systemctl stop ceph-mds(a)mds01.service
mds01:~ # cephfs-journal-tool journal export myjournal.bin
mds01:~ # cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary
mds01:~ # cephfs-journal-tool --rank=cephfs:0 journal reset
mds01:~ # cephfs-table-tool all reset session
mds01:~ # systemctl start ceph-mds(a)mds01.service
mds01:~ # ceph mds repaired 0
mds01:~ # ceph daemon mds.mds01 scrub_path / recursive repair
2. Do the same recovery for the MSD.1 as in step 1 and
bring MDS.1 also up.
If step 1 succeeds the standby daemons will most likely also start
successfully.
Zitat von Sagara Wijetunga <sagarawmw(a)yahoo.com>om>:
> Sorry, above post has to be corrected as:"Out of the info now
> emerged so far seems Ceph client wanted to write an object of
> size 1555896 but managed to write only 1540096 bytes to the journal."
>
> Sagara
> On Saturday, May 22, 2021, 08:29:34 PM GMT+8, Sagara Wijetunga
> <sagarawmw(a)yahoo.com> wrote:
>
> Out of the info now emerged so far seems Ceph client wanted to
> write an object of size 1555896 but managed to write only 1555896
> bytes to the journal.
> I think what we need to do now is:1. Get the MDS.0 recover, discard
> if necessary part of the object 200.00006048 and bring the MSD.0 up.
2. Do the same recovery for the MSD.1 as in step 1 and
bring MDS.1 also up.
> 3. Above two steps to the most probability may bring
CephFS up.
> 4. Once the CephFS is up, scan for corrupted files, remove them and
> bring from backup.
> 5. Get the MDS.2 to sync to MSD.0 or 1 and bring the cluster to
> sync'ed stage.
>
> My question is, what exactly necessary to carry above step 1?
> Sagara
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
4:29 p.m.
Hi Eugen
Now the Ceph is HEALTH_OK.
I think what we need to do now is:
1. Get the MDS.0 recover, discard if necessary part of the object
200.00006048 and bring the MSD.0 up.
Yes, I agree, I just can't tell what the best way is here, maybe
remove all three objects from the disks (make a backup before doing
that, just in case) and try the steps to recover the journal (also
make a backup of the journal first):
mds01:~ # systemctl stop ceph-mds(a)mds01.service
mds01:~ # cephfs-journal-tool journal export myjournal.bin
mds01:~ # cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary
mds01:~ # cephfs-journal-tool --rank=cephfs:0 journal reset
mds01:~ # cephfs-table-tool all reset session
mds01:~ # systemctl start ceph-mds(a)mds01.service
mds01:~ # ceph mds repaired 0
mds01:~ # ceph daemon mds.mds01 scrub_path / recursive repair
Only the last step above failed as follows:
# ceph daemon mds.a scrub_path / recursive repair
"mds_not_active"
failed
But the ceph -w showed:
2021-05-22 23:30:00.199164 mon.a [INF] Health check cleared: MDS_DAMAGE (was: 1 mds daemon
damaged)
2021-05-22 23:30:00.208558 mon.a [INF] Standby daemon mds.c assigned to filesystem cephfs
as rank 0
2021-05-22 23:30:00.208614 mon.a [INF] Health check cleared: MDS_ALL_DOWN (was: 1
filesystem is offline)
2021-05-22 23:30:04.029282 mon.a [INF] daemon mds.c is now active in filesystem cephfs as
rank 0
2021-05-22 23:30:04.378670 mon.a [INF] Health check cleared: FS_DEGRADED (was: 1
filesystem is degraded)
Since most errors fixed, I tried to repair 2.44:
ceph pg repair 2.44
ceph -w
2021-05-23 00:00:00.009926 mon.a [ERR] overall HEALTH_ERR 4 scrub errors; Possible data
damage: 1 pg inconsistent
2021-05-23 00:01:17.454975 mon.a [INF] Health check cleared: OSD_SCRUB_ERRORS (was: 4
scrub errors)
2021-05-23 00:01:17.454993 mon.a [INF] Health check cleared: PG_DAMAGED (was: Possible
data damage: 1 pg inconsistent)
2021-05-23 00:01:17.455002 mon.a [INF] Cluster is now healthy
2021-05-23 00:01:13.544097 osd.0 [ERR] 2.44 repair : stat mismatch, got 108/109 objects,
0/0 clones, 108/109 dirty, 108/109 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts,
0/1555896 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.
2021-05-23 00:01:13.544154 osd.0 [ERR] 2.44 repair 1 errors, 1 fixed
# ceph -s
cluster:
id: abc...
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 22h)
mgr: a(active, since 22h), standbys: b, c
mds: cephfs:1 {0=c=up:active} 2 up:standby
osd: 3 osds: 3 up (since 22h), 3 in (since 22h)
task status:
scrub status:
mds.c: idle
data: pools: 3 pools, 192 pgs
objects: 281.06k objects, 327 GiB
usage: 2.4 TiB used, 8.1 TiB / 11 TiB avail
pgs: 192 active+clean
I mounted the CephFS as before and tried following:cephfs-data-scan pg_files
/mnt/ceph/Home/sagara 2.44
But it complains invalid path. I'm trying to see what files are effected by the
missing object in PG 2.44.
Thank you very much helping this far.
But I still prefer to understand whether any file effected by this disaster.
Best regardsSagara
5:10 p.m.
Awesome! I'm glad it worked out this far! At least you have a working
filesystem now even it means that you may have to use a backup.
But now I can say it: Having only three OSDs is really not the best
idea. ;-) Are all those OSDs on the same host?
Zitat von Sagara Wijetunga <sagarawmw(a)yahoo.com>om>:
Hi Eugen
Now the Ceph is HEALTH_OK.
I think what we need to do now is:
1. Get the MDS.0 recover, discard if necessary part of the object
200.00006048 and bring the MSD.0 up.
Yes, I agree, I just can't tell what the best way is here, maybe
remove all three objects from the disks (make a backup before doing
that, just in case) and try the steps to recover the journal (also
make a backup of the journal first):
mds01:~ # systemctl stop ceph-mds(a)mds01.service
mds01:~ # cephfs-journal-tool journal export myjournal.bin
mds01:~ # cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary
mds01:~ # cephfs-journal-tool --rank=cephfs:0 journal reset
mds01:~ # cephfs-table-tool all reset session
mds01:~ # systemctl start ceph-mds(a)mds01.service
mds01:~ # ceph mds repaired 0
mds01:~ # ceph daemon mds.mds01 scrub_path / recursive repair
Only the last step above failed as follows:
# ceph daemon mds.a scrub_path / recursive repair
"mds_not_active"
failed
But the ceph -w showed:
2021-05-22 23:30:00.199164 mon.a [INF] Health check cleared:
MDS_DAMAGE (was: 1 mds daemon damaged)
2021-05-22 23:30:00.208558 mon.a [INF] Standby daemon mds.c assigned
to filesystem cephfs as rank 0
2021-05-22 23:30:00.208614 mon.a [INF] Health check cleared:
MDS_ALL_DOWN (was: 1 filesystem is offline)
2021-05-22 23:30:04.029282 mon.a [INF] daemon mds.c is now active in
filesystem cephfs as rank 0
2021-05-22 23:30:04.378670 mon.a [INF] Health check cleared:
FS_DEGRADED (was: 1 filesystem is degraded)
Since most errors fixed, I tried to repair 2.44:
ceph pg repair 2.44
ceph -w
2021-05-23 00:00:00.009926 mon.a [ERR] overall HEALTH_ERR 4 scrub
errors; Possible data damage: 1 pg inconsistent
2021-05-23 00:01:17.454975 mon.a [INF] Health check cleared:
OSD_SCRUB_ERRORS (was: 4 scrub errors)
2021-05-23 00:01:17.454993 mon.a [INF] Health check cleared:
PG_DAMAGED (was: Possible data damage: 1 pg inconsistent)
2021-05-23 00:01:17.455002 mon.a [INF] Cluster is now healthy
2021-05-23 00:01:13.544097 osd.0 [ERR] 2.44 repair : stat mismatch,
got 108/109 objects, 0/0 clones, 108/109 dirty, 108/109 omap, 0/0
pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/1555896 bytes, 0/0
manifest objects, 0/0 hit_set_archive bytes.
2021-05-23 00:01:13.544154 osd.0 [ERR] 2.44 repair 1 errors, 1 fixed
# ceph -s
cluster:
id: abc...
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 22h)
mgr: a(active, since 22h), standbys: b, c
mds: cephfs:1 {0=c=up:active} 2 up:standby
osd: 3 osds: 3 up (since 22h), 3 in (since 22h)
task status:
scrub status:
mds.c: idle
data: pools: 3 pools, 192 pgs
objects: 281.06k objects, 327 GiB
usage: 2.4 TiB used, 8.1 TiB / 11 TiB avail
pgs: 192 active+clean
I mounted the CephFS as before and tried following:cephfs-data-scan
pg_files /mnt/ceph/Home/sagara 2.44
But it complains invalid path. I'm trying to see what files are
effected by the missing object in PG 2.44.
Thank you very much helping this far.
But I still prefer to understand whether any file effected by this disaster.
Best regardsSagara
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
23 May
23 May
4:45 a.m.
On Sunday, May 23, 2021, 01:16:12 AM GMT+8, Eugen Block <eblock(a)nde.ag> wrote:
Awesome! I'm glad it worked out this far! At least you have a working
filesystem now even it means that you may have to use a backup.
But now I can say it: Having only three OSDs is really not the best
idea. ;-) Are all those OSDs on the same host?
1. For safe side I did a full deep-scrubceph osd deep-scrub all
ceph -w shows no error, only following line repeating:2021-05-23 01:00:00.003140 mon.a
[INF] overall HEALTH_OK
2021-05-23 02:00:00.007661 mon.a [INF] overall HEALTH_OK
That is, whatever in the cluster is clean.
2. I take daily rsync-based backups.
I still not sure what the removed metadata object represented.
3. I have allocated three (3) separate machines for the Ceph cluster. That is, I have 3
separate instances of MON, MGR, OSD and MDS running on 3 separate machines.
I agree it is better allocate five (5) different machines with pool size 5. It further
reduces the risk factor that losing quorum if one machine is already down.
I think to avoid this kind of mess happening again, have to use data center-grade SSDs
with PLP (Power Loss Protection). Mine is hard disks.
The issue with data center-grade SSDs with PLP is still low capacity and very expensive.
One not so expensive option is to keep the journal on separate data center-grade SSD with
PLP. But Ceph has to give me a guarantee that it's flushing or sync the journal to
high capacity hard disks are fail safe. What's your understanding on this? Is it fail
safe? Any link for me to further read.
Best regardsSagara
8:31 a.m.
Hi,
3. I have allocated three (3) separate machines for
the Ceph
cluster. That is, I have 3 separate instances of MON, MGR, OSD and
MDS running on 3 separate machines.
okay, so at least those are three different hosts, although in a
production environment I would strongly recommend to use a dedicated
MDS server. But why only three OSDs? In case of a disk failure the
cluster is in a degraded state until you recover or rebuild that one
OSD on that host. If you had more disks per node those PGs could at
least be remapped to a different OSD and let the cluster recover.
The other thing is to have the CephFS metadata pool on SSDs, that's a
common recommendation to reduce latency. And since the metadata pool
is usually quite small it wouldn't be that expensive.
Increasing the number of MONs to 5 is not unreasonable although most
of our customers (as well as our own cluster) are fine with 3 MONs.
But increasing the pool size to 5 can or will have an impact on the
performance since it also increases the latency, every write has to be
acked 5 times instead of 3. I think you'd be fine with pool size 3
(failure domain host) but you should move the metadata to SSDs and
increase the overall number of OSDs.
There is no guarantee, you can only reduce the risks of data loss but
prepare for it with backups.
[5] https://docs.ceph.com/en/latest/cephfs/createfs/
Zitat von Sagara Wijetunga <sagarawmw(a)yahoo.com>om>:
> On Sunday, May 23, 2021, 01:16:12 AM GMT+8, Eugen Block
> <eblock(a)nde.ag> wrote: Awesome! I'm glad it worked out this far! At
> least you have a working
> filesystem now even it means that you may have to use a backup.
> But now I can say it: Having only three OSDs is really not the best
> idea. ;-) Are all those OSDs on the same host?
>
> 1. For safe side I did a full deep-scrubceph osd deep-scrub all
>
> ceph -w shows no error, only following line repeating:2021-05-23
> 01:00:00.003140 mon.a [INF] overall HEALTH_OK
>
> 2021-05-23 02:00:00.007661 mon.a [INF] overall HEALTH_OK
>
> That is, whatever in the cluster is clean.
>
>
> 2. I take daily rsync-based backups.
> I still not sure what the removed metadata object represented.
>
>
3. I have allocated three (3) separate machines for
the Ceph
cluster. That is, I have 3 separate instances of MON, MGR, OSD and
MDS running on 3 separate machines.
> I agree it is better allocate five (5)
different machines with pool
> size 5. It further reduces the risk factor that losing quorum if one
> machine is already down.
> I think to avoid this kind of mess happening again, have to use data
> center-grade SSDs with PLP (Power Loss Protection). Mine is hard
> disks.
> The issue with data center-grade SSDs with PLP is still low capacity
> and very expensive.
> One not so expensive option is to keep the journal on separate data
> center-grade SSD with PLP. But Ceph has to give me a guarantee that
> it's flushing or sync the journal to high capacity hard disks are
> fail safe. What's your understanding on this? Is it fail safe? Any
> link for me to further read.
> Best regardsSagara
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:48 a.m.
Also what is the status of you other mds ?
Is it active ? Or which one was damaged?
U can also look for additional mds in the same cluster possibility?
On Sat, 22 May 2021, 00:40 Eugen Block, <eblock(a)nde.ag> wrote:
Hi,
I went through similar trouble just this week [1], but the root cause
seems different so it probably won't apply to your case.
Which version of ceph are you running? There are a couple of reports
with similar error messages, e. g. [2], it may already been resolved.
Can you share
rados list-inconsistent-obj 2.44
and
ceph tell mds.<MDS> damage ls
The pool size is 3, right?
Regards,
Eugen
Zitat von Sagara Wijetunga <sagarawmw(a)yahoo.com>om>:
Hi all
An accidental power failure happened.
That resulted CephFS offline and cannot be mounted.
I have 3 MDS daemons but it complains "1 mds daemon damaged".
It seems a PG of cephfs_metadata is inconsistent. I tried to repair,
but doesn't get it repaired.
How do I repair the damaged MDS and bring the CephFS up/online?
Details are included below.
Many thanks in advance.
Sagara
# ceph -s
cluster:
id: abc...
health: HEALTH_ERR
1 filesystem is degraded
1 filesystem is offline
1 mds daemon damaged
4 scrub errors
Possible data damage: 1 pg inconsistent
services:
mon: 3 daemons, quorum a,b,c (age 107s)
mgr: a(active, since 22m), standbys: b, c
mds: cephfs:0/1 3 up:standby, 1 damaged
osd: 3 osds: 3 up (since 96s), 3 in (since 96s)
data:
pools: 3 pools, 192 pgs
objects: 281.05k objects, 327 GiB
usage: 2.4 TiB used, 8.1 TiB / 11 TiB avail
pgs: 191 active+clean
1 active+clean+inconsistent
# ceph health detail
HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds
daemon damaged; 4 scrub errors; Possible data damage: 1 pg
inconsistent
FS_DEGRADED 1 filesystem is degraded
fs cephfs is degraded
MDS_ALL_DOWN 1 filesystem is offline
fs cephfs is offline because no MDS is active for it.
MDS_DAMAGE 1 mds daemon damaged
fs cephfs mds.0 is damaged
OSD_SCRUB_ERRORS 4 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
pg 2.44 is active+clean+inconsistent, acting [0,2,1]
# ceph osd lspools
2 cephfs_metadata
3 cephfs_data
4 rbd
# ceph pg repair 2.44
# ceph -w
2021-05-22 01:48:04.775783 osd.0 [ERR] 2.44 shard 0 soid
2:22efaf6a:::200.00006048:head : candidate size 1540096 info size
1555896 mismatch
2021-05-22 01:48:04.775786 osd.0 [ERR] 2.44 shard 1 soid
2:22efaf6a:::200.00006048:head : candidate size 1540096 info size
1555896 mismatch
2021-05-22 01:48:04.775787 osd.0 [ERR] 2.44 shard 2 soid
2:22efaf6a:::200.00006048:head : candidate size 1441792 info size
1555896 mismatch
2021-05-22 01:48:04.775789 osd.0 [ERR] 2.44 soid
2:22efaf6a:::200.00006048:head : failed to pick suitable object info
2021-05-22 01:48:04.775849 osd.0 [ERR] repair 2.44
2:22efaf6a:::200.00006048:head : on disk size (1540096) does not
match object info size (1555896) adjusted for ondisk to (1555896)
2021-05-22 01:48:04.787167 osd.0 [ERR] 2.44 repair 4 errors, 0 fixed
--- End of detail ---
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1073
days inactive
1075
days old
15 comments
3 participants
participants (3)
-
Eugen Block -
Lokendra Rathour -
Sagara Wijetunga