Ceph Filesystem recovery with intact pools
Hi, I've had a complete monitor failure, which I have recovered from with the steps here: https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/#... The data and metadata pools are there and are completely intact, but ceph is reporting that there are no filesystems, where (before the failure) there was one. Is there any way of putting the filesystem back together again without having to resort to having to rebuild a new metadata pool with cephfs-data-scan? I'm on ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Thanks, Harlan
Hi, how exactly does ceph report that there’s no CephFS? If your MONs were down and you recovered them, is at least one MGR also up and running? Can you share ‚ceph -s‘ and ‚ceph fs status’? Zitat von cyclic3.git@gmail.com:
Hi, I've had a complete monitor failure, which I have recovered from with the steps here: https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/#... The data and metadata pools are there and are completely intact, but ceph is reporting that there are no filesystems, where (before the failure) there was one.
Is there any way of putting the filesystem back together again without having to resort to having to rebuild a new metadata pool with cephfs-data-scan? I'm on ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
Thanks, Harlan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
My ceph -s output is this: cluster: id: bfe08dcf-aabd-4cac-ac4f-9e56af3df11b health: HEALTH_ERR 1/3 mons down, quorum omicron-m1,omicron-m2 6 scrub errors Possible data damage: 1 pg inconsistent Degraded data redundancy: 626702/20558920 objects degraded (3.048%), 32 pgs degraded, 32 pgs undersized 94 daemons have recently crashed services: mon: 3 daemons, quorum omicron-m1,omicron-m2 (age 8h), out of quorum: omicron-m0 mgr: omicron-m0(active, since 28h) osd: 33 osds: 32 up (since 28h), 32 in (since 28h) data: pools: 8 pools, 736 pgs objects: 9.97M objects, 9.9 TiB usage: 21 TiB used, 27 TiB / 47 TiB avail pgs: 626702/20558920 objects degraded (3.048%) 702 active+clean 32 active+undersized+degraded 1 active+clean+inconsistent 1 active+clean+scrubbing+deep+repair and ceph fs status prints nothing.
There’s no MDS running, can you start it? Zitat von cyclic3.git@gmail.com:
My ceph -s output is this: cluster: id: bfe08dcf-aabd-4cac-ac4f-9e56af3df11b health: HEALTH_ERR 1/3 mons down, quorum omicron-m1,omicron-m2 6 scrub errors Possible data damage: 1 pg inconsistent Degraded data redundancy: 626702/20558920 objects degraded (3.048%), 32 pgs degraded, 32 pgs undersized 94 daemons have recently crashed
services: mon: 3 daemons, quorum omicron-m1,omicron-m2 (age 8h), out of quorum: omicron-m0 mgr: omicron-m0(active, since 28h) osd: 33 osds: 32 up (since 28h), 32 in (since 28h)
data: pools: 8 pools, 736 pgs objects: 9.97M objects, 9.9 TiB usage: 21 TiB used, 27 TiB / 47 TiB avail pgs: 626702/20558920 objects degraded (3.048%) 702 active+clean 32 active+undersized+degraded 1 active+clean+inconsistent 1 active+clean+scrubbing+deep+repair
and ceph fs status prints nothing. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I added an MDS, but there was no change in either output (apart from recognising the existence of an MDS)
I don’t understand, what happened to the previous MDS? If there are cephfs pools there also was an old MDS, right? Can you explain that please? Zitat von cyclic3.git@gmail.com:
I added an MDS, but there was no change in either output (apart from recognising the existence of an MDS) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Both the MDS maps and the keyrings are lost as a side effect of the monitor recovery process I mentioned in my initial email, detailed here https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/#... . On Mon, 31 Aug 2020 at 21:10, Eugen Block <eblock@nde.ag> wrote:
I don’t understand, what happened to the previous MDS? If there are cephfs pools there also was an old MDS, right? Can you explain that please?
Zitat von cyclic3.git@gmail.com:
I added an MDS, but there was no change in either output (apart from recognising the existence of an MDS) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Alright, I didn't realize that the MDS was affected by this as well. In that case there's probably no other way than running the 'ceph fs new ...' command as Yan, Zheng suggested. Do you have backups of your cephfs contents in case that goes wrong? I'm not sure if a pool copy would help in any way here, also I haven't recreated a cephfs from existing pools yet, maybe someone else can provide some more details about the risks of doing that, I understand your hesitation though. Regards, Eugen Zitat von Cyclic 3 <cyclic3.git@gmail.com>:
Both the MDS maps and the keyrings are lost as a side effect of the monitor recovery process I mentioned in my initial email, detailed here https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/#... .
On Mon, 31 Aug 2020 at 21:10, Eugen Block <eblock@nde.ag> wrote:
I don’t understand, what happened to the previous MDS? If there are cephfs pools there also was an old MDS, right? Can you explain that please?
Zitat von cyclic3.git@gmail.com:
I added an MDS, but there was no change in either output (apart from recognising the existence of an MDS) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Sun, Aug 30, 2020 at 8:05 PM <cyclic3.git@gmail.com> wrote:
Hi, I've had a complete monitor failure, which I have recovered from with the steps here: https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/#... The data and metadata pools are there and are completely intact, but ceph is reporting that there are no filesystems, where (before the failure) there was one.
Is there any way of putting the filesystem back together again without having to resort to having to rebuild a new metadata pool with cephfs-data-scan? I'm on ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
'ceph fs new <fs_name> <metadata> <data> [--force] [--allow-dangerous-metadata-overlay]' 'ceph fs new' command can create fs using existing pools. before running the command, make sure there is no mds running. after run the "fs new "command, run 'ceph fs reset <fs name> --yes-i-really-mean-it' immediately.
Thanks, Harlan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This sounds rather risky; will this definitely not lose any of my data?
With that first command, I get this error: Error EINVAL: pool 'cephfs_metadata' already contains some objects. Use an empty pool instead. What can I do?
participants (4)
-
Cyclic 3
-
cyclic3.git＠gmail.com
-
Eugen Block
-
Yan, Zheng