Hi Patrick, this was all on version 17.2.7. The mon store had to be rebuilt from OSDs, so the MDS map got lost. After recovering the ceph cluster itself we inspected the journal and it reported missing objects before we continued with the disaster recovery, this was the output (I had posted it in a different thread in the ceph-users mailing list before creating this thread): ---snip--- cephfs-journal-tool --rank=storage:0 --journal=mdlog journal inspect 023-12-08T15:35:22.922+0200 7f834d0320c0 -1 Missing object 200.000527c4 2023-12-08T15:35:22.938+0200 7f834d0320c0 -1 Bad entry start ptr (0x149f140067f) at 0x149f1174595 2023-12-08T15:35:22.942+0200 7f834d0320c0 -1 Bad entry start ptr (0x149f1400e66) at 0x149f1174d7c 2023-12-08T15:35:22.954+0200 7f834d0320c0 -1 Bad entry start ptr (0x149f1401642) at 0x149f1175558 2023-12-08T15:35:22.970+0200 7f834d0320c0 -1 Bad entry start ptr (0x149f1401e29) at 0x149f1175d3f 2023-12-08T15:35:22.974+0200 7f834d0320c0 -1 Bad entry start ptr (0x149f1402610) at 0x149f1176526 2023-12-08T15:35:22.978+0200 7f834d0320c0 -1 Missing object 200.000527ca 2023-12-08T15:35:22.978+0200 7f834d0320c0 -1 Missing object 200.000527cb 2023-12-08T15:35:22.994+0200 7f834d0320c0 -1 Bad entry start ptr (0x149f30008f4) at 0x149f2d7480a 2023-12-08T15:35:22.998+0200 7f834d0320c0 -1 Bad entry start ptr (0x149f3000ced) at 0x149f2d74c03 Overall journal integrity: DAMAGED Objects missing: 0x527c4 0x527ca 0x527cb Corrupt regions: 0x149f0d73f16-149f1174595 0x149f1174595-149f1174d7c 0x149f1174d7c-149f1175558 0x149f1175558-149f1175d3f 0x149f1175d3f-149f1176526 0x149f1176526-149f2d7480a 0x149f2d7480a-149f2d74c03 0x149f2d74c03-ffffffffffffffff cephfs-journal-tool --rank=storage:0 --journal=purge_queue journal inspect 2023-12-08T15:35:57.691+0200 7f331621e0c0 -1 Missing object 500.00000dc6 Overall journal integrity: DAMAGED Objects missing: 0xdc6 Corrupt regions: 0x3718522e9-ffffffffffffffff ---snip--- Zitat von Patrick Donnelly <pdonnell@redhat.com>:
On Mon, Dec 11, 2023 at 6:38 AM Eugen Block <eblock@nde.ag> wrote:
Hi,
I'm trying to help someone with a broken CephFS. We managed to recover basic ceph functionality but the CephFS is still inaccessible (currently read-only). We went through the disaster recovery steps but to no avail. Here's a snippet from the startup logs:
---snip--- mds.0.41 Booting: 2: waiting for purge queue recovered mds.0.journaler.pq(ro) _finish_probe_end write_pos = 14797504512 (header had 14789452521). recovered. mds.0.purge_queue operator(): open complete mds.0.purge_queue operator(): recovering write_pos monclient: get_auth_request con 0x55c280bc5c00 auth_method 0 monclient: get_auth_request con 0x55c280ee0c00 auth_method 0 mds.0.journaler.pq(ro) _finish_read got error -2 mds.0.purge_queue _recover: Error -2 recovering write_pos mds.0.purge_queue _go_readonly: going readonly because internal IO failed: No such file or directory mds.0.journaler.pq(ro) set_readonly mds.0.41 unhandled write error (2) No such file or directory, force readonly... mds.0.cache force file system read-only force file system read-only ---snip---
I've added the dev mailing list, maybe someone can give some advice how to continue from here (we could try to recover with an empty metadata pool). Or is this FS lost?
Looks like one of the purge queue journal objects was lost? Were other objects lost? It would be helpful to know more about the circumstances of this "broken CephFS"? What Ceph version?
-- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D