24 Feb
2020
24 Feb
'20
3:09 a.m.
I have a 3 node ceph cluster for my house that I have been using for a few
years now without issue. Each node is a MON, MGR, and MDS, and has 2-3 OSDs
on them. It has, however been slow. I decided to finally move the bluestore
DBs to SSDs. I did one OSD as a test case to make sure everything was going
to go OK. I deleted the OSD, then created a new OSD using the ceph-deploy
tool and pointed the DB at a LVM partition on a SSD.
Everything went OK, and recovery started. Later in the day I noticed that
my MDS daemon is damaged (PGs are still recovering).
I've tried the cephfs-journal-tool --rank=cephfs:all journal export
backup.bin command, but it gave me:
2020-02-23 17:50:03.589 7f7d8b225740 -1 Missing object 200.00c30b6d
2020-02-23 17:50:07.919 7f7d8b225740 -1 Bad entry start ptr
(0x30c2dbb92003) at 0x30c2d3a125ea
(both lines have several repeats)
and will not complete.
Looking at the log file of the mds that was active at the time shows:
2020-02-23 17:13:09.091 7fad40029700 0 mds.0.journaler.mdlog(ro)
_finish_read got error -2
2020-02-23 17:13:09.091 7fad40029700 0 mds.0.journaler.mdlog(ro)
_finish_read got error -2
2020-02-23 17:13:09.091 7fad40029700 0 mds.0.journaler.mdlog(ro)
_finish_read got error -2
2020-02-23 17:13:09.091 7fad3e826700 0 mds.0.log _replay journaler got
error -2, aborting
2020-02-23 17:13:09.091 7fad3e826700 -1 log_channel(cluster) log [ERR] :
missing journal object
One other thing that happened about the same time, I noticed I was having
memory pressure on all the nodes with only 200MB of free ram. I've tweaked
the bluestore osd_memory_target to try to help that not happen again. Even
so, I'm a bit confused how that could cause catastrophic failure, as I had
2 other MDSes on standby.
Any help would be appreciated.