[ceph-users] One mds daemon damaged, filesystem is offline. How to recover?

Hi all An accidental power failure happened. That resulted CephFS offline and cannot be mounted. I have 3 MDS daemons but it complains "1 mds daemon damaged". It seems a PG of cephfs_metadata is inconsistent. I tried to repair, but doesn't get it repaired. How do I repair the damaged MDS and bring the CephFS up/online? Details are included below. Many thanks in advance. Sagara # ceph -s   cluster:     id:     abc...     health: HEALTH_ERR             1 filesystem is degraded             1 filesystem is offline             1 mds daemon damaged             4 scrub errors             Possible data damage: 1 pg inconsistent     services:     mon: 3 daemons, quorum a,b,c (age 107s)     mgr: a(active, since 22m), standbys: b, c     mds: cephfs:0/1 3 up:standby, 1 damaged     osd: 3 osds: 3 up (since 96s), 3 in (since 96s)     data:     pools:   3 pools, 192 pgs     objects: 281.05k objects, 327 GiB     usage:   2.4 TiB used, 8.1 TiB / 11 TiB avail     pgs:     191 active+clean              1   active+clean+inconsistent # ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 1 mds daemon damaged; 4 scrub errors; Possible data damage: 1 pg inconsistent FS_DEGRADED 1 filesystem is degraded     fs cephfs is degraded MDS_ALL_DOWN 1 filesystem is offline     fs cephfs is offline because no MDS is active for it. MDS_DAMAGE 1 mds daemon damaged     fs cephfs mds.0 is damaged OSD_SCRUB_ERRORS 4 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent     pg 2.44 is active+clean+inconsistent, acting [0,2,1] # ceph osd lspools 2 cephfs_metadata 3 cephfs_data 4 rbd # ceph pg repair 2.44 # ceph -w 2021-05-22 01:48:04.775783 osd.0 [ERR] 2.44 shard 0 soid 2:22efaf6a:::200.00006048:head : candidate size 1540096 info size 1555896 mismatch 2021-05-22 01:48:04.775786 osd.0 [ERR] 2.44 shard 1 soid 2:22efaf6a:::200.00006048:head : candidate size 1540096 info size 1555896 mismatch 2021-05-22 01:48:04.775787 osd.0 [ERR] 2.44 shard 2 soid 2:22efaf6a:::200.00006048:head : candidate size 1441792 info size 1555896 mismatch 2021-05-22 01:48:04.775789 osd.0 [ERR] 2.44 soid 2:22efaf6a:::200.00006048:head : failed to pick suitable object info 2021-05-22 01:48:04.775849 osd.0 [ERR] repair 2.44 2:22efaf6a:::200.00006048:head : on disk size (1540096) does not match object info size (1555896) adjusted for ondisk to (1555896) 2021-05-22 01:48:04.787167 osd.0 [ERR] 2.44 repair 4 errors, 0 fixed --- End of detail ---