Urgent help: I accidentally nuked all my Monitor
Hello everyone, I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup. I am very grateful for all help! Best regards, Jonas Schwab
Hi Jonas, Am 4/10/25 um 16:01 schrieb Jonas Schwab:
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
There is a procedure to recover the MON-DB from the OSDs: https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#rec... Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Thank you very much! I now stated the first step, namely "Collect the map from each OSD host". As I have a cephadm deployment, I will have to execute ceph-objectstore-tool within each container. Unfortunately, this produces the error "Mount failed with '(11) Resource temporarily unavailable'". Does anybody know how to solve this? Best regards, Jonas On 2025-04-10 16:04, Robert Sander wrote:
Hi Jonas,
Am 4/10/25 um 16:01 schrieb Jonas Schwab:
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
There is a procedure to recover the MON-DB from the OSDs:
https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#rec...
Regards
Did you stop the OSDs? Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Thank you very much! I now stated the first step, namely "Collect the map from each OSD host". As I have a cephadm deployment, I will have to execute ceph-objectstore-tool within each container. Unfortunately, this produces the error "Mount failed with '(11) Resource temporarily unavailable'". Does anybody know how to solve this?
Best regards, Jonas
On 2025-04-10 16:04, Robert Sander wrote:
Hi Jonas,
Am 4/10/25 um 16:01 schrieb Jonas Schwab:
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
There is a procedure to recover the MON-DB from the OSDs:
https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#rec...
Regards
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
No, didn't issue any commands to the OSDs. On 2025-04-10 17:28, Eugen Block wrote:
Did you stop the OSDs?
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Thank you very much! I now stated the first step, namely "Collect the map from each OSD host". As I have a cephadm deployment, I will have to execute ceph-objectstore-tool within each container. Unfortunately, this produces the error "Mount failed with '(11) Resource temporarily unavailable'". Does anybody know how to solve this?
Best regards, Jonas
On 2025-04-10 16:04, Robert Sander wrote:
Hi Jonas,
Am 4/10/25 um 16:01 schrieb Jonas Schwab:
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
There is a procedure to recover the MON-DB from the OSDs:
https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#rec...
Regards
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460
You have to stop the OSDs in order to mount them with the objectstore tool. Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
No, didn't issue any commands to the OSDs.
On 2025-04-10 17:28, Eugen Block wrote:
Did you stop the OSDs?
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Thank you very much! I now stated the first step, namely "Collect the map from each OSD host". As I have a cephadm deployment, I will have to execute ceph-objectstore-tool within each container. Unfortunately, this produces the error "Mount failed with '(11) Resource temporarily unavailable'". Does anybody know how to solve this?
Best regards, Jonas
On 2025-04-10 16:04, Robert Sander wrote:
Hi Jonas,
Am 4/10/25 um 16:01 schrieb Jonas Schwab:
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
There is a procedure to recover the MON-DB from the OSDs:
https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#rec...
Regards
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you for the help! Does that mean stopping the container and mounting the lv? On 2025-04-10 17:38, Eugen Block wrote:
You have to stop the OSDs in order to mount them with the objectstore tool.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
No, didn't issue any commands to the OSDs.
On 2025-04-10 17:28, Eugen Block wrote:
Did you stop the OSDs?
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Thank you very much! I now stated the first step, namely "Collect the map from each OSD host". As I have a cephadm deployment, I will have to execute ceph-objectstore-tool within each container. Unfortunately, this produces the error "Mount failed with '(11) Resource temporarily unavailable'". Does anybody know how to solve this?
Best regards, Jonas
On 2025-04-10 16:04, Robert Sander wrote:
Hi Jonas,
Am 4/10/25 um 16:01 schrieb Jonas Schwab:
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
There is a procedure to recover the MON-DB from the OSDs:
https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#rec...
Regards
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460
No, you have to run the objectstore-tool command within the cephadm shell: cephadm shell --name osd.x -- ceph-objectstore-tool <rest of the command> There are plenty examples online. I’m on my mobile phone right now Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Thank you for the help! Does that mean stopping the container and mounting the lv?
On 2025-04-10 17:38, Eugen Block wrote:
You have to stop the OSDs in order to mount them with the objectstore tool.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
No, didn't issue any commands to the OSDs.
On 2025-04-10 17:28, Eugen Block wrote:
Did you stop the OSDs?
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Thank you very much! I now stated the first step, namely "Collect the map from each OSD host". As I have a cephadm deployment, I will have to execute ceph-objectstore-tool within each container. Unfortunately, this produces the error "Mount failed with '(11) Resource temporarily unavailable'". Does anybody know how to solve this?
Best regards, Jonas
On 2025-04-10 16:04, Robert Sander wrote:
Hi Jonas,
Am 4/10/25 um 16:01 schrieb Jonas Schwab:
> I believe I accidentally nuked all monitor of my cluster (please > don't > ask how). Is there a way to recover from this desaster? I have a > cephadm > setup.
There is a procedure to recover the MON-DB from the OSDs:
https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-mon/#rec...
Regards
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Can you bring back at least one of them? In that case you could reduce the monmap to 1 mon and bring the cluster back up. If the MONs are really dead, you can recover using OSDs [0]. I've never had to use that myself, but people have reported that to work. [0] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#r... Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Hello everyone,
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
I am very grateful for all help!
Best regards, Jonas Schwab _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 4/10/25 10:01 AM, Jonas Schwab wrote:
Hello everyone,
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster?
Depends on how really “nuked.” Are there monitor directories with data still under /var/lib/ceph/<fsid> by a chance? If so, monitors can be started simply as ceph-mon services, at least temporarily, by pointing to those directories. -- Šarūnas Burdulis Dartmouth Mathematics math.dartmouth.edu/~sarunas · https://useplaintext.email ·
I realized, I have access to a data directory of a monitor I removed just before the oopsie happened. Can I launch a ceph-mon from that? If I try just to launch ceph-mon, it commits suicide: 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 not in monmap and have been in a quorum before; must have been removed 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 commit suicide! 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 failed to initialize On 2025-04-10 16:01, Jonas Schwab wrote:
Hello everyone,
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
I am very grateful for all help!
Best regards, Jonas Schwab _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460
It can work, but it might be necessary to modify the monmap first, since it's complaining that it has been removed from it. Are you familiar with the monmap-tool (https://docs.ceph.com/en/latest/man/8/monmaptool/)? The procedure is similar to changing a monitor's IP address the "messy way" (https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-...). I also wrote a blog post how to do it with cephadm: https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm-changing-a-monitors-... But before changing anything, I'd inspect first what the current status is. You can get the current monmap from within the mon container (is it still there?): cephadm shell --name mon.<mon> ceph-monstore-tool /var/lib/ceph/mon/<your_mon> get monmap -- --out monmap monmaptool --print monmap You can paste the output here, if you want. Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
I realized, I have access to a data directory of a monitor I removed just before the oopsie happened. Can I launch a ceph-mon from that? If I try just to launch ceph-mon, it commits suicide:
2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 not in monmap and have been in a quorum before; must have been removed 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 commit suicide! 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 failed to initialize
On 2025-04-10 16:01, Jonas Schwab wrote:
Hello everyone,
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
I am very grateful for all help!
Best regards, Jonas Schwab _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Again, thank you very much for your help! The container is not there any more, but I discovered that the "old" mon data still exists. I have the same situation for two mons I removed at the same time: $ monmaptool --print monmap1 monmaptool: monmap file monmap1 epoch 29 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:21.203171+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (squid) election_strategy: 1 0: [v2:10.127.239.2:3300/0,v1:10.127.239.2:6789/0] mon.ceph2-02 1: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 2: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 3: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05 $ monmaptool --print monmap2 monmaptool: monmap file monmap2 epoch 30 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:43.216713+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (unknown) election_strategy: 1 0: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 1: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 2: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05 Would it be feasible to move the data from node1 (which still contains node2 as mon) to node2, or would that just result in even more mess? On 2025-04-10 19:57, Eugen Block wrote:
It can work, but it might be necessary to modify the monmap first, since it's complaining that it has been removed from it. Are you familiar with the monmap-tool (https://docs.ceph.com/en/latest/man/8/monmaptool/)?
The procedure is similar to changing a monitor's IP address the "messy way" (https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-...).
I also wrote a blog post how to do it with cephadm: https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm-changing-a-monitors-...
But before changing anything, I'd inspect first what the current status is. You can get the current monmap from within the mon container (is it still there?):
cephadm shell --name mon.<mon> ceph-monstore-tool /var/lib/ceph/mon/<your_mon> get monmap -- --out monmap monmaptool --print monmap
You can paste the output here, if you want.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
I realized, I have access to a data directory of a monitor I removed just before the oopsie happened. Can I launch a ceph-mon from that? If I try just to launch ceph-mon, it commits suicide:
2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 not in monmap and have been in a quorum before; must have been removed 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 commit suicide! 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 failed to initialize
On 2025-04-10 16:01, Jonas Schwab wrote:
Hello everyone,
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
I am very grateful for all help!
Best regards, Jonas Schwab _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It depends a bit. Which mon do the OSDs still know about? You can check /var/lib/ceph/<ceph_fsid>/osd.X/config to retrieve that piece of information. I'd try to revive one of them. Do you still have the mon store.db for all of the mons or at least one of them? Just to be safe, back up all the store.db directories. Then modify a monmap to contain the one you want to revive by removing the other ones. Backup your monmap files as well. Then inject the modified monmap into the daemon and try starting it. Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Again, thank you very much for your help!
The container is not there any more, but I discovered that the "old" mon data still exists. I have the same situation for two mons I removed at the same time:
$ monmaptool --print monmap1 monmaptool: monmap file monmap1 epoch 29 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:21.203171+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (squid) election_strategy: 1 0: [v2:10.127.239.2:3300/0,v1:10.127.239.2:6789/0] mon.ceph2-02 1: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 2: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 3: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05
$ monmaptool --print monmap2 monmaptool: monmap file monmap2 epoch 30 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:43.216713+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (unknown) election_strategy: 1 0: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 1: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 2: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05
Would it be feasible to move the data from node1 (which still contains node2 as mon) to node2, or would that just result in even more mess?
On 2025-04-10 19:57, Eugen Block wrote:
It can work, but it might be necessary to modify the monmap first, since it's complaining that it has been removed from it. Are you familiar with the monmap-tool (https://docs.ceph.com/en/latest/man/8/monmaptool/)?
The procedure is similar to changing a monitor's IP address the "messy way" (https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-...).
I also wrote a blog post how to do it with cephadm: https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm-changing-a-monitors-...
But before changing anything, I'd inspect first what the current status is. You can get the current monmap from within the mon container (is it still there?):
cephadm shell --name mon.<mon> ceph-monstore-tool /var/lib/ceph/mon/<your_mon> get monmap -- --out monmap monmaptool --print monmap
You can paste the output here, if you want.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
I realized, I have access to a data directory of a monitor I removed just before the oopsie happened. Can I launch a ceph-mon from that? If I try just to launch ceph-mon, it commits suicide:
2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 not in monmap and have been in a quorum before; must have been removed 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 commit suicide! 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 failed to initialize
On 2025-04-10 16:01, Jonas Schwab wrote:
Hello everyone,
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
I am very grateful for all help!
Best regards, Jonas Schwab _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I edited the monmap to include only rgw2-06 and then followed https://docs.ceph.com/en/squid/rados/operations/add-or-rm-mons/#adding-a-mon... to create a new monitor. Unfortunately, `ceph-mon -i mon.rgw2-06 --public-addr 10.127.239.63 -f` crashed with the traceback seen in the attachment. On 2025-04-10 20:34, Eugen Block wrote:
It depends a bit. Which mon do the OSDs still know about? You can check /var/lib/ceph/<ceph_fsid>/osd.X/config to retrieve that piece of information. I'd try to revive one of them. Do you still have the mon store.db for all of the mons or at least one of them? Just to be safe, back up all the store.db directories.
Then modify a monmap to contain the one you want to revive by removing the other ones. Backup your monmap files as well. Then inject the modified monmap into the daemon and try starting it.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Again, thank you very much for your help!
The container is not there any more, but I discovered that the "old" mon data still exists. I have the same situation for two mons I removed at the same time:
$ monmaptool --print monmap1 monmaptool: monmap file monmap1 epoch 29 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:21.203171+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (squid) election_strategy: 1 0: [v2:10.127.239.2:3300/0,v1:10.127.239.2:6789/0] mon.ceph2-02 1: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 2: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 3: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05
$ monmaptool --print monmap2 monmaptool: monmap file monmap2 epoch 30 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:43.216713+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (unknown) election_strategy: 1 0: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 1: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 2: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05
Would it be feasible to move the data from node1 (which still contains node2 as mon) to node2, or would that just result in even more mess?
On 2025-04-10 19:57, Eugen Block wrote:
It can work, but it might be necessary to modify the monmap first, since it's complaining that it has been removed from it. Are you familiar with the monmap-tool (https://docs.ceph.com/en/latest/man/8/monmaptool/)?
The procedure is similar to changing a monitor's IP address the "messy way" (https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-...).
I also wrote a blog post how to do it with cephadm: https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm-changing-a-monitors-...
But before changing anything, I'd inspect first what the current status is. You can get the current monmap from within the mon container (is it still there?):
cephadm shell --name mon.<mon> ceph-monstore-tool /var/lib/ceph/mon/<your_mon> get monmap -- --out monmap monmaptool --print monmap
You can paste the output here, if you want.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
I realized, I have access to a data directory of a monitor I removed just before the oopsie happened. Can I launch a ceph-mon from that? If I try just to launch ceph-mon, it commits suicide:
2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 not in monmap and have been in a quorum before; must have been removed 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 commit suicide! 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 failed to initialize
On 2025-04-10 16:01, Jonas Schwab wrote:
Hello everyone,
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
I am very grateful for all help!
Best regards, Jonas Schwab _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460
I solved the problem with executing ceph-mon. Among others, -i mon.rgw2-06 was not the correct option, but rather -i rgw2-06. Unfortunately, that brought the next problem: The cluster now shows "100.000% pgs unknown", which is probably because the monitor data is not complete up to date, but rather the state it was in before I switched over to other mons. A few minutes or s after that, the cluster crashed and I lust the mons. I guess this outdated cluster map is probably unusable? All services seem to be running fine and there are not network obstructions. Should I instead go with this: https://docs.ceph.com/en/squid/rados/troubleshooting/troubleshooting-mon/#re... ? I actually already tried the latter option, but ran into the error `rocksdb: [db/db_impl/db_impl_open.cc:2086] DB::Open() failed: IO error: while open a file for lock: /var/lib/ceph/mon/ceph-ceph2-01/store.db/LOCK: Permission denied` Even though I double checked that the permission and ownership on the replacing store.db are properly set. On 2025-04-10 22:45, Jonas Schwab wrote:
I edited the monmap to include only rgw2-06 and then followed https://docs.ceph.com/en/squid/rados/operations/add-or-rm-mons/#adding-a-mon...
to create a new monitor.
Unfortunately, `ceph-mon -i mon.rgw2-06 --public-addr 10.127.239.63 -f` crashed with the traceback seen in the attachment.
On 2025-04-10 20:34, Eugen Block wrote:
It depends a bit. Which mon do the OSDs still know about? You can check /var/lib/ceph/<ceph_fsid>/osd.X/config to retrieve that piece of information. I'd try to revive one of them. Do you still have the mon store.db for all of the mons or at least one of them? Just to be safe, back up all the store.db directories.
Then modify a monmap to contain the one you want to revive by removing the other ones. Backup your monmap files as well. Then inject the modified monmap into the daemon and try starting it.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Again, thank you very much for your help!
The container is not there any more, but I discovered that the "old" mon data still exists. I have the same situation for two mons I removed at the same time:
$ monmaptool --print monmap1 monmaptool: monmap file monmap1 epoch 29 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:21.203171+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (squid) election_strategy: 1 0: [v2:10.127.239.2:3300/0,v1:10.127.239.2:6789/0] mon.ceph2-02 1: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 2: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 3: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05
$ monmaptool --print monmap2 monmaptool: monmap file monmap2 epoch 30 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:43.216713+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (unknown) election_strategy: 1 0: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 1: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 2: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05
Would it be feasible to move the data from node1 (which still contains node2 as mon) to node2, or would that just result in even more mess?
On 2025-04-10 19:57, Eugen Block wrote:
It can work, but it might be necessary to modify the monmap first, since it's complaining that it has been removed from it. Are you familiar with the monmap-tool (https://docs.ceph.com/en/latest/man/8/monmaptool/)?
The procedure is similar to changing a monitor's IP address the "messy way" (https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-...).
I also wrote a blog post how to do it with cephadm: https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm-changing-a-monitors-...
But before changing anything, I'd inspect first what the current status is. You can get the current monmap from within the mon container (is it still there?):
cephadm shell --name mon.<mon> ceph-monstore-tool /var/lib/ceph/mon/<your_mon> get monmap -- --out monmap monmaptool --print monmap
You can paste the output here, if you want.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
I realized, I have access to a data directory of a monitor I removed just before the oopsie happened. Can I launch a ceph-mon from that? If I try just to launch ceph-mon, it commits suicide:
2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 not in monmap and have been in a quorum before; must have been removed 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 commit suicide! 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 failed to initialize
On 2025-04-10 16:01, Jonas Schwab wrote:
Hello everyone,
I believe I accidentally nuked all monitor of my cluster (please don't ask how). Is there a way to recover from this desaster? I have a cephadm setup.
I am very grateful for all help!
Best regards, Jonas Schwab _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Is at least one mgr running? PG states are reported by the mgr daemon. Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
I solved the problem with executing ceph-mon. Among others, -i mon.rgw2-06 was not the correct option, but rather -i rgw2-06. Unfortunately, that brought the next problem:
The cluster now shows "100.000% pgs unknown", which is probably because the monitor data is not complete up to date, but rather the state it was in before I switched over to other mons. A few minutes or s after that, the cluster crashed and I lust the mons. I guess this outdated cluster map is probably unusable? All services seem to be running fine and there are not network obstructions.
Should I instead go with this: https://docs.ceph.com/en/squid/rados/troubleshooting/troubleshooting-mon/#re... ?
I actually already tried the latter option, but ran into the error `rocksdb: [db/db_impl/db_impl_open.cc:2086] DB::Open() failed: IO error: while open a file for lock: /var/lib/ceph/mon/ceph-ceph2-01/store.db/LOCK: Permission denied` Even though I double checked that the permission and ownership on the replacing store.db are properly set.
On 2025-04-10 22:45, Jonas Schwab wrote:
I edited the monmap to include only rgw2-06 and then followed https://docs.ceph.com/en/squid/rados/operations/add-or-rm-mons/#adding-a-mon...
to create a new monitor.
Unfortunately, `ceph-mon -i mon.rgw2-06 --public-addr 10.127.239.63 -f` crashed with the traceback seen in the attachment.
On 2025-04-10 20:34, Eugen Block wrote:
It depends a bit. Which mon do the OSDs still know about? You can check /var/lib/ceph/<ceph_fsid>/osd.X/config to retrieve that piece of information. I'd try to revive one of them. Do you still have the mon store.db for all of the mons or at least one of them? Just to be safe, back up all the store.db directories.
Then modify a monmap to contain the one you want to revive by removing the other ones. Backup your monmap files as well. Then inject the modified monmap into the daemon and try starting it.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Again, thank you very much for your help!
The container is not there any more, but I discovered that the "old" mon data still exists. I have the same situation for two mons I removed at the same time:
$ monmaptool --print monmap1 monmaptool: monmap file monmap1 epoch 29 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:21.203171+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (squid) election_strategy: 1 0: [v2:10.127.239.2:3300/0,v1:10.127.239.2:6789/0] mon.ceph2-02 1: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 2: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 3: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05
$ monmaptool --print monmap2 monmaptool: monmap file monmap2 epoch 30 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:43.216713+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (unknown) election_strategy: 1 0: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 1: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 2: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05
Would it be feasible to move the data from node1 (which still contains node2 as mon) to node2, or would that just result in even more mess?
On 2025-04-10 19:57, Eugen Block wrote:
It can work, but it might be necessary to modify the monmap first, since it's complaining that it has been removed from it. Are you familiar with the monmap-tool (https://docs.ceph.com/en/latest/man/8/monmaptool/)?
The procedure is similar to changing a monitor's IP address the "messy way" (https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-...).
I also wrote a blog post how to do it with cephadm: https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm-changing-a-monitors-...
But before changing anything, I'd inspect first what the current status is. You can get the current monmap from within the mon container (is it still there?):
cephadm shell --name mon.<mon> ceph-monstore-tool /var/lib/ceph/mon/<your_mon> get monmap -- --out monmap monmaptool --print monmap
You can paste the output here, if you want.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
I realized, I have access to a data directory of a monitor I removed just before the oopsie happened. Can I launch a ceph-mon from that? If I try just to launch ceph-mon, it commits suicide:
2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 not in monmap and have been in a quorum before; must have been removed 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 mon.mon.ceph2-01@-1(???) e29 commit suicide! 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 failed to initialize
On 2025-04-10 16:01, Jonas Schwab wrote: > Hello everyone, > > I believe I accidentally nuked all monitor of my cluster (please > don't > ask how). Is there a way to recover from this desaster? I have a > cephadm > setup. > > I am very grateful for all help! > > Best regards, > Jonas Schwab > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes mgrs are running as intended. It just seems that mons and osd don't recongnize each other, because the monitors map is outdated. On 2025-04-11 07:07, Eugen Block wrote:
Is at least one mgr running? PG states are reported by the mgr daemon.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
I solved the problem with executing ceph-mon. Among others, -i mon.rgw2-06 was not the correct option, but rather -i rgw2-06. Unfortunately, that brought the next problem:
The cluster now shows "100.000% pgs unknown", which is probably because the monitor data is not complete up to date, but rather the state it was in before I switched over to other mons. A few minutes or s after that, the cluster crashed and I lust the mons. I guess this outdated cluster map is probably unusable? All services seem to be running fine and there are not network obstructions.
Should I instead go with this: https://docs.ceph.com/en/squid/rados/troubleshooting/troubleshooting-mon/#re...
?
I actually already tried the latter option, but ran into the error `rocksdb: [db/db_impl/db_impl_open.cc:2086] DB::Open() failed: IO error: while open a file for lock: /var/lib/ceph/mon/ceph-ceph2-01/store.db/LOCK: Permission denied` Even though I double checked that the permission and ownership on the replacing store.db are properly set.
On 2025-04-10 22:45, Jonas Schwab wrote:
I edited the monmap to include only rgw2-06 and then followed https://docs.ceph.com/en/squid/rados/operations/add-or-rm-mons/#adding-a-mon...
to create a new monitor.
Unfortunately, `ceph-mon -i mon.rgw2-06 --public-addr 10.127.239.63 -f` crashed with the traceback seen in the attachment.
On 2025-04-10 20:34, Eugen Block wrote:
It depends a bit. Which mon do the OSDs still know about? You can check /var/lib/ceph/<ceph_fsid>/osd.X/config to retrieve that piece of information. I'd try to revive one of them. Do you still have the mon store.db for all of the mons or at least one of them? Just to be safe, back up all the store.db directories.
Then modify a monmap to contain the one you want to revive by removing the other ones. Backup your monmap files as well. Then inject the modified monmap into the daemon and try starting it.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Again, thank you very much for your help!
The container is not there any more, but I discovered that the "old" mon data still exists. I have the same situation for two mons I removed at the same time:
$ monmaptool --print monmap1 monmaptool: monmap file monmap1 epoch 29 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:21.203171+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (squid) election_strategy: 1 0: [v2:10.127.239.2:3300/0,v1:10.127.239.2:6789/0] mon.ceph2-02 1: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 2: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 3: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05
$ monmaptool --print monmap2 monmaptool: monmap file monmap2 epoch 30 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:43.216713+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (unknown) election_strategy: 1 0: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 1: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 2: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05
Would it be feasible to move the data from node1 (which still contains node2 as mon) to node2, or would that just result in even more mess?
On 2025-04-10 19:57, Eugen Block wrote:
It can work, but it might be necessary to modify the monmap first, since it's complaining that it has been removed from it. Are you familiar with the monmap-tool (https://docs.ceph.com/en/latest/man/8/monmaptool/)?
The procedure is similar to changing a monitor's IP address the "messy way" (https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-...).
I also wrote a blog post how to do it with cephadm: https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm-changing-a-monitors-...
But before changing anything, I'd inspect first what the current status is. You can get the current monmap from within the mon container (is it still there?):
cephadm shell --name mon.<mon> ceph-monstore-tool /var/lib/ceph/mon/<your_mon> get monmap -- --out monmap monmaptool --print monmap
You can paste the output here, if you want.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
> I realized, I have access to a data directory of a monitor I > removed > just before the oopsie happened. Can I launch a ceph-mon from that? > If I > try just to launch ceph-mon, it commits suicide: > > 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 > mon.mon.ceph2-01@-1(???) > e29 not in monmap and have been in a quorum before; must have been > removed > 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 > mon.mon.ceph2-01@-1(???) > e29 commit suicide! > 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 failed to initialize > > On 2025-04-10 16:01, Jonas Schwab wrote: >> Hello everyone, >> >> I believe I accidentally nuked all monitor of my cluster (please >> don't >> ask how). Is there a way to recover from this desaster? I have a >> cephadm >> setup. >> >> I am very grateful for all help! >> >> Best regards, >> Jonas Schwab >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > -- > Jonas Schwab > > Research Data Management, Cluster of Excellence ct.qmat > https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de > Email: jonas.schwab@uni-wuerzburg.de > Tel: +49 931 31-84460 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460
For anyone interested, I offered Jonas my assistance off list. After he already had collected the cluster maps from all the OSDs and rebuilt the MON store, the three MONs did start, but failed probing/electing. We then decided to reduce the monmap to one MON only, and that worked, the cluster came back from the dead. After losing the mon store all kinds of manual configuration was necessary, like setting public_network again, enable the orchestrator, setting the cephadm user + private/public key, import keyrings, etc. etc. Jonas did all of that by himself, and after the orchestrator was back, he was able to deploy two MONs again. He also got more RAM (one of the issues in the other thread) and now his OSDs are all fine again. He also managed to bring back his CephFS. Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Yes mgrs are running as intended. It just seems that mons and osd don't recongnize each other, because the monitors map is outdated.
On 2025-04-11 07:07, Eugen Block wrote:
Is at least one mgr running? PG states are reported by the mgr daemon.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
I solved the problem with executing ceph-mon. Among others, -i mon.rgw2-06 was not the correct option, but rather -i rgw2-06. Unfortunately, that brought the next problem:
The cluster now shows "100.000% pgs unknown", which is probably because the monitor data is not complete up to date, but rather the state it was in before I switched over to other mons. A few minutes or s after that, the cluster crashed and I lust the mons. I guess this outdated cluster map is probably unusable? All services seem to be running fine and there are not network obstructions.
Should I instead go with this: https://docs.ceph.com/en/squid/rados/troubleshooting/troubleshooting-mon/#re...
?
I actually already tried the latter option, but ran into the error `rocksdb: [db/db_impl/db_impl_open.cc:2086] DB::Open() failed: IO error: while open a file for lock: /var/lib/ceph/mon/ceph-ceph2-01/store.db/LOCK: Permission denied` Even though I double checked that the permission and ownership on the replacing store.db are properly set.
On 2025-04-10 22:45, Jonas Schwab wrote:
I edited the monmap to include only rgw2-06 and then followed https://docs.ceph.com/en/squid/rados/operations/add-or-rm-mons/#adding-a-mon...
to create a new monitor.
Unfortunately, `ceph-mon -i mon.rgw2-06 --public-addr 10.127.239.63 -f` crashed with the traceback seen in the attachment.
On 2025-04-10 20:34, Eugen Block wrote:
It depends a bit. Which mon do the OSDs still know about? You can check /var/lib/ceph/<ceph_fsid>/osd.X/config to retrieve that piece of information. I'd try to revive one of them. Do you still have the mon store.db for all of the mons or at least one of them? Just to be safe, back up all the store.db directories.
Then modify a monmap to contain the one you want to revive by removing the other ones. Backup your monmap files as well. Then inject the modified monmap into the daemon and try starting it.
Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>:
Again, thank you very much for your help!
The container is not there any more, but I discovered that the "old" mon data still exists. I have the same situation for two mons I removed at the same time:
$ monmaptool --print monmap1 monmaptool: monmap file monmap1 epoch 29 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:21.203171+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (squid) election_strategy: 1 0: [v2:10.127.239.2:3300/0,v1:10.127.239.2:6789/0] mon.ceph2-02 1: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 2: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 3: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05
$ monmaptool --print monmap2 monmaptool: monmap file monmap2 epoch 30 fsid 6d0d4ed4-0052-4eb9-9d9d-e6872ba7ee96 last_changed 2025-04-10T14:16:43.216713+0200 created 2021-02-26T14:02:29.522695+0100 min_mon_release 19 (unknown) election_strategy: 1 0: [v2:10.127.239.61:3300/0,v1:10.127.239.61:6789/0] mon.rgw2-04 1: [v2:10.127.239.63:3300/0,v1:10.127.239.63:6789/0] mon.rgw2-06 2: [v2:10.127.239.62:3300/0,v1:10.127.239.62:6789/0] mon.rgw2-05
Would it be feasible to move the data from node1 (which still contains node2 as mon) to node2, or would that just result in even more mess?
On 2025-04-10 19:57, Eugen Block wrote: > It can work, but it might be necessary to modify the monmap first, > since it's complaining that it has been removed from it. Are you > familiar with the monmap-tool > (https://docs.ceph.com/en/latest/man/8/monmaptool/)? > > The procedure is similar to changing a monitor's IP address the > "messy > way" > (https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#changing-a-...). > > > > > > I also wrote a blog post how to do it with cephadm: > https://heiterbiswolkig.blogs.nde.ag/2020/12/18/cephadm-changing-a-monitors-... > > > > > > But before changing anything, I'd inspect first what the current > status is. You can get the current monmap from within the mon > container (is it still there?): > > cephadm shell --name mon.<mon> > ceph-monstore-tool /var/lib/ceph/mon/<your_mon> get monmap -- --out > monmap > monmaptool --print monmap > > You can paste the output here, if you want. > > Zitat von Jonas Schwab <jonas.schwab@uni-wuerzburg.de>: > >> I realized, I have access to a data directory of a monitor I >> removed >> just before the oopsie happened. Can I launch a ceph-mon from that? >> If I >> try just to launch ceph-mon, it commits suicide: >> >> 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 >> mon.mon.ceph2-01@-1(???) >> e29 not in monmap and have been in a quorum before; must have been >> removed >> 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 >> mon.mon.ceph2-01@-1(???) >> e29 commit suicide! >> 2025-04-10T19:32:32.174+0200 7fec628c5e00 -1 failed to initialize >> >> On 2025-04-10 16:01, Jonas Schwab wrote: >>> Hello everyone, >>> >>> I believe I accidentally nuked all monitor of my cluster (please >>> don't >>> ask how). Is there a way to recover from this desaster? I have a >>> cephadm >>> setup. >>> >>> I am very grateful for all help! >>> >>> Best regards, >>> Jonas Schwab >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> -- >> Jonas Schwab >> >> Research Data Management, Cluster of Excellence ct.qmat >> https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de >> Email: jonas.schwab@uni-wuerzburg.de >> Tel: +49 931 31-84460 >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Jonas Schwab
Research Data Management, Cluster of Excellence ct.qmat https://data.ctqmat.de | datamanagement.ct.qmat@listserv.dfn.de Email: jonas.schwab@uni-wuerzburg.de Tel: +49 931 31-84460 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Eugen Block
-
Jonas Schwab
-
Robert Sander
-
Šarūnas Burdulis