Hi, I'm not familiar with rook so the steps required may vary. If you try to reuse the old mon stores you'll have the mentioned mismatch between the new daemons and the old monmap (which still contains the old mon daemons). It's not entirely clear what went wrong in the first place and what you already tried exactly, so it's hard to tell if editing the monmap is the way to go here. I guess the old mon daemons are removed, is that assumption correct? In that case it could be worth a try to edit the current monmap to contain only the new mons and inject it (see [1] for details). If the mons start and form a quorum you'd have a cluster, but I can't tell if the OSDs will register successfully. I think the previous approach when the original mons were up but the OSDs didn't start would have been more promising. Anyway, maybe editing the monmap will fix this for you. [1] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#r... Zitat von Ben <ruidong.gao@gmail.com>:
Hi Eugen,
Thank you for help on this.
Forget the log. A little progress, the monitors store were restored. I created a new ceph cluster to use the restored monitors store. But the monitor log complains:
debug 2023-03-09T11:00:31.233+0000 7fe95234f880 0 starting mon.a rank -1 at public addrs [v2:169.169.163.25:3300/0,v1:169.169.163.25:6789/0] at bind addrs [v2:197.166.206.27:3300/0,v1:197.166.206.27:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3f271841-6188-47c1-b3fd-90fd4f978c76
debug 2023-03-09T11:00:31.234+0000 7fe95234f880 1 mon.a@-1(???) e27 preinit fsid 3f271841-6188-47c1-b3fd-90fd4f978c76
debug 2023-03-09T11:00:31.234+0000 7fe95234f880 -1 mon.a@-1(???) e27 not in monmap and have been in a quorum before; must have been removed
debug 2023-03-09T11:00:31.234+0000 7fe95234f880 -1 mon.a@-1(???) e27 commit suicide!
debug 2023-03-09T11:00:31.234+0000 7fe95234f880 -1 failed to initialize
The fact is original monitor clusters ids are k,m,o, however the new ones are a,b,d. It was deployed by rook. Any ideas to make this work?
Ben
Eugen Block <eblock@nde.ag> 于2023年3月9日周四 16:00写道：
Hi,
there's no attachment to your email, please use something like pastebin to provide OSD logs.
Thanks Eugen
Zitat von Ben <ruidong.gao@gmail.com>:
Hi,
I ended up with having whole set of osds to get back original ceph cluster. I figured out to make the cluster running. However, it's status is something as below:
bash-4.4$ ceph -s
cluster:
id: 3f271841-6188-47c1-b3fd-90fd4f978c76
health: HEALTH_WARN
7 daemons have recently crashed
4 slow ops, oldest one blocked for 35077 sec, daemons [mon.a,mon.b] have slow ops.
services:
mon: 3 daemons, quorum a,b,d (age 9h)
mgr: b(active, since 14h), standbys: a
osd: 4 osds: 0 up, 4 in (since 9h)
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
All osds are down.
I checked the osds logs and attached with this.
Please help and I wonder if it's possible to get the cluster back. I have some backup for monitor's data. Till now I haven't restore that in the course.
Thanks,
Ben _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io