Sorry, forgot the links: [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/4WJ5IUANKYO4... [2] https://tracker.ceph.com/issues/43068 Zitat von Eugen Block <eblock@nde.ag>:

Hi,

I went through similar trouble just this week [1], but the root cause seems different so it probably won't apply to your case. Which version of ceph are you running? There are a couple of reports with similar error messages, e. g. [2], it may already been resolved.

Can you share

rados list-inconsistent-obj 2.44

and

ceph tell mds.<MDS> damage ls

The pool size is 3, right?

Regards, Eugen

Zitat von Sagara Wijetunga <sagarawmw@yahoo.com>:

...

Hi all An accidental power failure happened. That resulted CephFS offline and cannot be mounted. I have 3 MDS daemons but it complains "1 mds daemon damaged".

It seems a PG of cephfs_metadata is inconsistent. I tried to repair, but doesn't get it repaired. How do I repair the damaged MDS and bring the CephFS up/online? Details are included below.

Many thanks in advance. Sagara

# ceph -s

  cluster:

    id:     abc...

    health: HEALTH_ERR

            1 filesystem is degraded

            1 filesystem is offline

            1 mds daemon damaged

            4 scrub errors

            Possible data damage: 1 pg inconsistent

 

  services:

    mon: 3 daemons, quorum a,b,c (age 107s)

    mgr: a(active, since 22m), standbys: b, c

    mds: cephfs:0/1 3 up:standby, 1 damaged

    osd: 3 osds: 3 up (since 96s), 3 in (since 96s)

 

  data:

    pools:   3 pools, 192 pgs

    objects: 281.05k objects, 327 GiB

    usage:   2.4 TiB used, 8.1 TiB / 11 TiB avail

    pgs:     191 active+clean

             1   active+clean+inconsistent

# ceph health detail

HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged; 4 scrub errors; Possible data damage: 1 pg inconsistent

FS_DEGRADED 1 filesystem is degraded

    fs cephfs is degraded

MDS_ALL_DOWN 1 filesystem is offline

    fs cephfs is offline because no MDS is active for it.

MDS_DAMAGE 1 mds daemon damaged

    fs cephfs mds.0 is damaged

OSD_SCRUB_ERRORS 4 scrub errors

PG_DAMAGED Possible data damage: 1 pg inconsistent

    pg 2.44 is active+clean+inconsistent, acting [0,2,1]

# ceph osd lspools

2 cephfs_metadata

3 cephfs_data

4 rbd

# ceph pg repair 2.44

# ceph -w

2021-05-22 01:48:04.775783 osd.0 [ERR] 2.44 shard 0 soid 2:22efaf6a:::200.00006048:head : candidate size 1540096 info size 1555896 mismatch

2021-05-22 01:48:04.775786 osd.0 [ERR] 2.44 shard 1 soid 2:22efaf6a:::200.00006048:head : candidate size 1540096 info size 1555896 mismatch

2021-05-22 01:48:04.775787 osd.0 [ERR] 2.44 shard 2 soid 2:22efaf6a:::200.00006048:head : candidate size 1441792 info size 1555896 mismatch

2021-05-22 01:48:04.775789 osd.0 [ERR] 2.44 soid 2:22efaf6a:::200.00006048:head : failed to pick suitable object info

2021-05-22 01:48:04.775849 osd.0 [ERR] repair 2.44 2:22efaf6a:::200.00006048:head : on disk size (1540096) does not match object info size (1555896) adjusted for ondisk to (1555896)

2021-05-22 01:48:04.787167 osd.0 [ERR] 2.44 repair 4 errors, 0 fixed

--- End of detail ---

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

What does the MDS report in its logs from when it went down? What size do you get when you run rados -p cephfs_metadata stat 200.00006048 There's a similar report [3] suggesting to try to force an update on the object info, you could give that a shot:

1. rados -p [cephfs_metadata] setomapval 200.00006048 temporary-key anything 2. ceph pg deep-scrub 2.44 3. Wait for the scrub to finish 4. rados -p [cephfs_metadata] rmomapkey 200.00006048 temporary-key

Above Ceph error messages shows "fs cephfs mds.0 is damaged". My MDS are named a,b and c. Does mds.0 means mds.a?

Here mds.0 means the rank 0, I would assume that you only have one rank, correct (one filesystem with standby MDS daemons)? [3] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-May/034580.html Zitat von Sagara Wijetunga <sagarawmw@yahoo.com>:

Hi Eugen Thanks for the reply. Ceph Version:# ceph versionceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)

...

Can you share

rados list-inconsistent-obj 2.44 # rados list-inconsistent-obj 2.44 {"epoch":6996,"inconsistents":[]}

...

ceph tell mds.<MDS> damage ls

# ceph tell mds.a damage ls 2021-05-22 13:23:34.135 80bf25c00  0 client.4344312 ms_handle_reset on v2:192.168.1.130:6810/3532878906 2021-05-22 13:23:34.146 80dcc2500  0 client.4344318 ms_handle_reset on v2:192.168.1.130:6810/3532878906 Error EINVAL: MDS not active

...

The pool size is 3, right?

Yes, pool size is 3.

Above Ceph error messages shows "fs cephfs mds.0 is damaged". My MDS are named a,b and c. Does mds.0 means mds.a? Since I have 2 MDS not damaged, how do I recover and mount the CephFS? Btw, our home directories are on CephFS, at the moment, no one can log in. Sagara

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Here are the physical file sizes of the "200.00006048*": OSD.0:-rw-r--r--  1 ceph  ceph  1540096 May 20 22:47 /var/lib/ceph/osd/ceph-0/current/2.44_head/200.00006048__head_56F5F744__ OSD.1:-rw-r--r--  1 ceph  ceph  1540096 May 20 22:47 /var/lib/ceph/osd/ceph-1/current/2.44_head/DIR_4/DIR_4/200.00006048__head_56F5F744__2 OSD2:-rw-r--r--  1 ceph  ceph  1441792 May 20 22:47 /var/lib/ceph/osd/ceph-2/current/2.44_head/DIR_4/DIR_4/200.00006048__head_56F5F744__2 Sagara

Out of the info now emerged so far seems Ceph client wanted to write an object of size 1555896 but managed to write only 1555896 bytes to the journal. I think what we need to do now is:1. Get the MDS.0 recover, discard if necessary part of the object 200.00006048 and bring the MSD.0 up. 2. Do the same recovery for the MSD.1 as in step 1 and bring MDS.1 also up. 3. Above two steps to the most probability may bring CephFS up. 4. Once the CephFS is up, scan for corrupted files, remove them and bring from backup. 5. Get the MDS.2 to sync to MSD.0 or 1 and bring the cluster to sync'ed stage. My question is, what exactly necessary to carry above step 1? Sagara

Sorry, above post has to be corrected as:"Out of the info now emerged so far seems Ceph client wanted to write an object of size 1555896 but managed to write only 1540096 bytes to the journal."

Yes, I would think so, too.

I think what we need to do now is: 1. Get the MDS.0 recover, discard if necessary part of the object 200.00006048 and bring the MSD.0 up.

Yes, I agree, I just can't tell what the best way is here, maybe remove all three objects from the disks (make a backup before doing that, just in case) and try the steps to recover the journal (also make a backup of the journal first): mds01:~ # systemctl stop ceph-mds@mds01.service mds01:~ # cephfs-journal-tool journal export myjournal.bin mds01:~ # cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary mds01:~ # cephfs-journal-tool --rank=cephfs:0 journal reset mds01:~ # cephfs-table-tool all reset session mds01:~ # systemctl start ceph-mds(a)mds01.service mds01:~ # ceph mds repaired 0 mds01:~ # ceph daemon mds.mds01 scrub_path / recursive repair

2. Do the same recovery for the MSD.1 as in step 1 and bring MDS.1 also up.

If step 1 succeeds the standby daemons will most likely also start successfully. Zitat von Sagara Wijetunga <sagarawmw@yahoo.com>:

Sorry, above post has to be corrected as:"Out of the info now emerged so far seems Ceph client wanted to write an object of size 1555896 but managed to write only  1540096 bytes to the journal."

Sagara On Saturday, May 22, 2021, 08:29:34 PM GMT+8, Sagara Wijetunga <sagarawmw@yahoo.com> wrote:

Out of the info now emerged so far seems Ceph client wanted to write an object of size 1555896 but managed to write only 1555896 bytes to the journal. I think what we need to do now is:1. Get the MDS.0 recover, discard if necessary part of the object 200.00006048 and bring the MSD.0 up. 2. Do the same recovery for the MSD.1 as in step 1 and bring MDS.1 also up. 3. Above two steps to the most probability may bring CephFS up. 4. Once the CephFS is up, scan for corrupted files, remove them and bring from backup. 5. Get the MDS.2 to sync to MSD.0 or 1 and bring the cluster to sync'ed stage.

My question is, what exactly necessary to carry above step 1? Sagara

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Awesome! I'm glad it worked out this far! At least you have a working filesystem now even it means that you may have to use a backup. But now I can say it: Having only three OSDs is really not the best idea. ;-) Are all those OSDs on the same host? Zitat von Sagara Wijetunga <sagarawmw@yahoo.com>:

Hi Eugen Now the Ceph is HEALTH_OK.

...

I think what we need to do now is: 1. Get the MDS.0 recover, discard if necessary part of the object  200.00006048 and bring the MSD.0 up.

Yes, I agree, I just can't tell what the best way is here, maybe  remove all three objects from the disks (make a backup before doing  that, just in case) and try the steps to recover the journal (also  make a backup of the journal first):

mds01:~ # systemctl stop ceph-mds@mds01.service

mds01:~ # cephfs-journal-tool journal export myjournal.bin

mds01:~ # cephfs-journal-tool --rank=cephfs:0 event recover_dentries summary

mds01:~ # cephfs-journal-tool --rank=cephfs:0 journal reset

mds01:~ # cephfs-table-tool all reset session

mds01:~ # systemctl start ceph-mds(a)mds01.service

mds01:~ # ceph mds repaired 0

mds01:~ # ceph daemon mds.mds01 scrub_path / recursive repair

Only the last step above failed as follows: # ceph daemon mds.a scrub_path / recursive repair "mds_not_active" failed

But the ceph -w showed: 2021-05-22 23:30:00.199164 mon.a [INF] Health check cleared: MDS_DAMAGE (was: 1 mds daemon damaged) 2021-05-22 23:30:00.208558 mon.a [INF] Standby daemon mds.c assigned to filesystem cephfs as rank 0 2021-05-22 23:30:00.208614 mon.a [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2021-05-22 23:30:04.029282 mon.a [INF] daemon mds.c is now active in filesystem cephfs as rank 0 2021-05-22 23:30:04.378670 mon.a [INF] Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded)

Since most errors fixed, I tried to repair 2.44: ceph pg repair 2.44 ceph -w 2021-05-23 00:00:00.009926 mon.a [ERR] overall HEALTH_ERR 4 scrub errors; Possible data damage: 1 pg inconsistent 2021-05-23 00:01:17.454975 mon.a [INF] Health check cleared: OSD_SCRUB_ERRORS (was: 4 scrub errors) 2021-05-23 00:01:17.454993 mon.a [INF] Health check cleared: PG_DAMAGED (was: Possible data damage: 1 pg inconsistent) 2021-05-23 00:01:17.455002 mon.a [INF] Cluster is now healthy 2021-05-23 00:01:13.544097 osd.0 [ERR] 2.44 repair : stat mismatch, got 108/109 objects, 0/0 clones, 108/109 dirty, 108/109 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/1555896 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes. 2021-05-23 00:01:13.544154 osd.0 [ERR] 2.44 repair 1 errors, 1 fixed

 # ceph -s   cluster:     id:     abc...     health: HEALTH_OK    services:     mon: 3 daemons, quorum a,b,c (age 22h)     mgr: a(active, since 22h), standbys: b, c     mds: cephfs:1 {0=c=up:active} 2 up:standby     osd: 3 osds: 3 up (since 22h), 3 in (since 22h)     task status:     scrub status:         mds.c: idle    data:    pools:   3 pools, 192 pgs     objects: 281.06k objects, 327 GiB     usage:   2.4 TiB used, 8.1 TiB / 11 TiB avail     pgs:     192 active+clean

I mounted the CephFS as before and tried following:cephfs-data-scan pg_files /mnt/ceph/Home/sagara 2.44

But it complains invalid path. I'm trying to see what files are effected by the missing object in PG 2.44. Thank you very much helping this far.  But I still prefer to understand whether any file effected by this disaster. Best regardsSagara

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi,

3. I have allocated three (3) separate machines for the Ceph cluster. That is, I have 3 separate instances of MON, MGR, OSD and MDS running on 3 separate machines.

okay, so at least those are three different hosts, although in a production environment I would strongly recommend to use a dedicated MDS server. But why only three OSDs? In case of a disk failure the cluster is in a degraded state until you recover or rebuild that one OSD on that host. If you had more disks per node those PGs could at least be remapped to a different OSD and let the cluster recover. The other thing is to have the CephFS metadata pool on SSDs, that's a common recommendation to reduce latency. And since the metadata pool is usually quite small it wouldn't be that expensive. Increasing the number of MONs to 5 is not unreasonable although most of our customers (as well as our own cluster) are fine with 3 MONs. But increasing the pool size to 5 can or will have an impact on the performance since it also increases the latency, every write has to be acked 5 times instead of 3. I think you'd be fine with pool size 3 (failure domain host) but you should move the metadata to SSDs and increase the overall number of OSDs. There is no guarantee, you can only reduce the risks of data loss but prepare for it with backups. [5] https://docs.ceph.com/en/latest/cephfs/createfs/ Zitat von Sagara Wijetunga <sagarawmw@yahoo.com>:

On Sunday, May 23, 2021, 01:16:12 AM GMT+8, Eugen Block <eblock@nde.ag> wrote: Awesome! I'm glad it worked out this far! At least you have a working  filesystem now even it means that you may have to use a backup. But now I can say it: Having only three OSDs is really not the best  idea. ;-) Are all those OSDs on the same host?

1. For safe side I did a full deep-scrubceph osd deep-scrub all

ceph -w shows no error, only following line repeating:2021-05-23 01:00:00.003140 mon.a [INF] overall HEALTH_OK

2021-05-23 02:00:00.007661 mon.a [INF] overall HEALTH_OK

That is, whatever in the cluster is clean.

2. I take daily rsync-based backups.  I still not sure what the removed metadata object represented. 

3. I have allocated three (3) separate machines for the Ceph cluster. That is, I have 3 separate instances of MON, MGR, OSD and MDS running on 3 separate machines. I agree it is better allocate five (5) different machines with pool size 5. It further reduces the risk factor that losing quorum if one machine is already down. I think to avoid this kind of mess happening again, have to use data center-grade SSDs with PLP (Power Loss Protection). Mine is hard disks.  The issue with data center-grade SSDs with PLP is still low capacity and very expensive. One not so expensive option is to keep the journal on separate data center-grade SSD with PLP. But Ceph has to give me a guarantee that it's flushing or sync the journal to high capacity hard disks are fail safe. What's your understanding on this? Is it fail safe? Any link for me to further read. Best regardsSagara

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io