24 Feb
2023
24 Feb
'23
4:20 p.m.
Hello all, I'd really appreciate some input from the more knowledgeable
here.
Is there a way I can access OSD objects if I have a BlueFS replay error?
This error prevents me starting the OSD and also throws an error if I try
using the bluestore or objectstore tools. - I can however run a
ceph-bluestore-tool show-label without issue.
I'm hoping there is another way or possibly a way to purge this log that I
can still access the objects on this OSD.
If deleting this reply log will help (even with some data loss) I'm happy
to try it.
This has caused a PG to go inactive and I'm considering deleting the PG and
force re-creating it. - Saw this mentioned as a last resort option.
Below is a snip of where things go wrong. - I don't know if there is even a
chance or is this an unrecoverable state?
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0:
op_dir_link db/031664.sst to 29549
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0:
op_dir_link db/031665.sst to 29550
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0:
op_dir_link db/031666.sst to 29551
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0:
op_dir_link db/CURRENT to 29543
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0:
op_dir_link db/IDENTITY to 5
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0:
op_dir_link db/LOCK to 2
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0:
op_dir_link db/MANIFEST-031657 to 29542
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0:
op_dir_link db/OPTIONS-031645 to 29529
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0:
op_dir_link db/OPTIONS-031660 to 29545
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0:
op_dir_create db.slow
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0: op_jump
seq 5204712 offset 0x20000
2023-01-25T10:05:26.543+0000 7fa773a14240 10 bluefs _read h 0x55d2f1cfdb80
0x10000~10000 from file(ino 1 size 0x0 mtime
2022-10-07T17:55:34.189440+0000 allocated 420000 alloc_commit 420000
extents [1:0x1770170000~20000,1:0x53d1e900000~400000])
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _read left 0x10000 len
0x10000
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _read got 65536
2023-01-25T10:05:26.543+0000 7fa773a14240 10 bluefs _read h 0x55d2f1cfdb80
0x20000~1000 from file(ino 1 size 0x20000 mtime
2022-10-07T17:55:34.189440+0000 allocated 420000 alloc_commit 420000
extents [1:0x1770170000~20000,1:0x53d1e900000~400000])
2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _read fetching
0x0~100000 of 1:0x53d1e900000~400000
2023-01-25T10:05:26.547+0000 7fa773a14240 20 bluefs _read left 0x100000 len
0x1000
2023-01-25T10:05:26.547+0000 7fa773a14240 20 bluefs _read got 4096
2023-01-25T10:05:26.547+0000 7fa773a14240 10 bluefs _replay 0x20000:
txn(seq 5204713 len 0x55 crc 0x81f48b1c)
2023-01-25T10:05:26.547+0000 7fa773a14240 20 bluefs _replay 0x20000:
op_file_update file(ino 29551 size 0x0 mtime
2022-10-07T17:55:34.151007+0000 allocated 0 alloc_commit 0 extents [])
2023-01-25T10:05:26.547+0000 7fa773a14240 20 bluefs _replay 0x20000:
op_dir_link db/031666.sst to 29551
2023-01-25T10:05:26.555+0000 7fa773a14240 -1
/build/ceph-17.2.5/src/os/bluestore/BlueFS.cc: In function 'int
BlueFS::_replay(bool, bool)' thread 7fa773a14240 time
2023-01-25T10:05:26.551808+0000
/build/ceph-17.2.5/src/os/bluestore/BlueFS.cc: 1419: FAILED ceph_assert(r
== q->second->file_map.end())
Kind regards
Geoff