30 Aug
2020
30 Aug
'20
4:34 p.m.
Hi,
I've had a complete monitor failure, which I have recovered from with the steps here:
https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/…
The data and metadata pools are there and are completely intact, but ceph is reporting
that there are no filesystems, where (before the failure) there was one.
Is there any way of putting the filesystem back together again without having to resort to
having to rebuild a new metadata pool with cephfs-data-scan?
I'm on ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus
(stable)
Thanks,
Harlan
30 Aug
30 Aug
11:53 p.m.
Hi,
how exactly does ceph report that there’s no CephFS? If your MONs were
down and you recovered them, is at least one MGR also up and running?
Can you share ‚ceph -s‘ and ‚ceph fs status’?
Zitat von cyclic3.git(a)gmail.com:
Hi,
I've had a complete monitor failure, which I have recovered from
with the steps here:
https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/…
The data and metadata pools are there and are completely intact, but
ceph is reporting that there are no filesystems, where (before the
failure) there was one.
Is there any way of putting the filesystem back together again
without having to resort to having to rebuild a new metadata pool
with cephfs-data-scan?
I'm on ceph version 15.2.4
(7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
Thanks,
Harlan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
31 Aug
31 Aug
1:08 a.m.
My ceph -s output is this:
cluster:
id: bfe08dcf-aabd-4cac-ac4f-9e56af3df11b
health: HEALTH_ERR
1/3 mons down, quorum omicron-m1,omicron-m2
6 scrub errors
Possible data damage: 1 pg inconsistent
Degraded data redundancy: 626702/20558920 objects degraded (3.048%), 32 pgs
degraded, 32 pgs undersized
94 daemons have recently crashed
services:
mon: 3 daemons, quorum omicron-m1,omicron-m2 (age 8h), out of quorum: omicron-m0
mgr: omicron-m0(active, since 28h)
osd: 33 osds: 32 up (since 28h), 32 in (since 28h)
data:
pools: 8 pools, 736 pgs
objects: 9.97M objects, 9.9 TiB
usage: 21 TiB used, 27 TiB / 47 TiB avail
pgs: 626702/20558920 objects degraded (3.048%)
702 active+clean
32 active+undersized+degraded
1 active+clean+inconsistent
1 active+clean+scrubbing+deep+repair
and ceph fs status prints nothing.
9:42 a.m.
There’s no MDS running, can you start it?
Zitat von cyclic3.git(a)gmail.com:
My ceph -s output is this:
cluster:
id: bfe08dcf-aabd-4cac-ac4f-9e56af3df11b
health: HEALTH_ERR
1/3 mons down, quorum omicron-m1,omicron-m2
6 scrub errors
Possible data damage: 1 pg inconsistent
Degraded data redundancy: 626702/20558920 objects
degraded (3.048%), 32 pgs degraded, 32 pgs undersized
94 daemons have recently crashed
services:
mon: 3 daemons, quorum omicron-m1,omicron-m2 (age 8h), out of
quorum: omicron-m0
mgr: omicron-m0(active, since 28h)
osd: 33 osds: 32 up (since 28h), 32 in (since 28h)
data:
pools: 8 pools, 736 pgs
objects: 9.97M objects, 9.9 TiB
usage: 21 TiB used, 27 TiB / 47 TiB avail
pgs: 626702/20558920 objects degraded (3.048%)
702 active+clean
32 active+undersized+degraded
1 active+clean+inconsistent
1 active+clean+scrubbing+deep+repair
and ceph fs status prints nothing.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:09 p.m.
I added an MDS, but there was no change in either output (apart from recognising the
existence of an MDS)
1 Sep
1 Sep
12:40 a.m.
I don’t understand, what happened to the previous MDS? If there are
cephfs pools there also was an old MDS, right? Can you explain that
please?
Zitat von cyclic3.git(a)gmail.com:
I added an MDS, but there was no change in either
output (apart from
recognising the existence of an MDS)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:03 a.m.
Both the MDS maps and the keyrings are lost as a side effect of the monitor
recovery process I mentioned in my initial email, detailed here
https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/…
.
On Mon, 31 Aug 2020 at 21:10, Eugen Block <eblock(a)nde.ag> wrote:
I don’t understand, what happened to the previous MDS?
If there are
cephfs pools there also was an old MDS, right? Can you explain that
please?
Zitat von cyclic3.git(a)gmail.com:
I added an MDS, but there was no change in either
output (apart from
recognising the existence of an MDS)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:51 a.m.
Alright, I didn't realize that the MDS was affected by this as well.
In that case there's probably no other way than running the 'ceph fs
new ...' command as Yan, Zheng suggested.
Do you have backups of your cephfs contents in case that goes wrong?
I'm not sure if a pool copy would help in any way here, also I haven't
recreated a cephfs from existing pools yet, maybe someone else can
provide some more details about the risks of doing that, I understand
your hesitation though.
Regards,
Eugen
Zitat von Cyclic 3 <cyclic3.git(a)gmail.com>om>:
Both the MDS maps and the keyrings are lost as a side
effect of the monitor
recovery process I mentioned in my initial email, detailed here
https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/…
.
On Mon, 31 Aug 2020 at 21:10, Eugen Block <eblock(a)nde.ag> wrote:
> I don’t understand, what happened to the previous MDS? If there are
> cephfs pools there also was an old MDS, right? Can you explain that
> please?
>
>
> Zitat von cyclic3.git(a)gmail.com:
>
> > I added an MDS, but there was no change in either output (apart from
> > recognising the existence of an MDS)
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
>
>
31 Aug
31 Aug
11:40 a.m.
On Sun, Aug 30, 2020 at 8:05 PM <cyclic3.git(a)gmail.com> wrote:
Hi,
I've had a complete monitor failure, which I have recovered from with the steps here:
https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/…
The data and metadata pools are there and are completely intact, but ceph is reporting
that there are no filesystems, where (before the failure) there was one.
Is there any way of putting the filesystem back together again without having to resort
to having to rebuild a new metadata pool with cephfs-data-scan?
I'm on ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus
(stable)
'ceph fs new <fs_name> <metadata> <data> [--force]
[--allow-dangerous-metadata-overlay]'
'ceph fs new' command can create fs using existing pools. before
running the command, make sure there is no mds running. after run the
"fs new "command, run 'ceph fs reset <fs name>
--yes-i-really-mean-it'
immediately.
> Thanks,
> Harlan
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
2 Sep
2 Sep
10:15 p.m.
With that first command, I get this error:
Error EINVAL: pool 'cephfs_metadata' already contains some objects. Use an empty
pool instead.
What can I do?
1326
days inactive
1329
days old
11 comments
4 participants
participants (4)
-
Cyclic 3 -
cyclic3.git＠gmail.com
-
Eugen Block -
Yan, Zheng