Hi all, I’m running a Ceph cluster managed by Rook on Kubernetes, and my CephFS metadata/journal appears to be in a bad state. I’d like to get advice before I attempt any destructive metadata repair operations. Below is a summary of the situation. *Environment* - Orchestrator: Rook/Ceph on Kubernetes (namespace: rook-ceph) - Ceph version: 18.2.2 (Reef, stable) for mon/mgr/osd/mds - Filesystem: - Name: rookfs - One metadata pool - One data pool *Cluster health* ceph status shows: - Health: HEALTH_WARN - Warnings: - 1 MDS reports slow metadata IOs - 1 MDS reports slow requests - Reduced data availability: some PGs in stale state - A large number of daemons have recently crashed MDS section reports: - 1/1 MDS daemons up, 1 hot standby CephFS state and MDS status ceph fs dump for rookfs shows - Filesystem rookfs is marked damaged. - max_mds = 1. - in set is empty. - up set is {0=<mds_id>}. - Flags mention allow_standby_replay. So, the filesystem is marked damaged in the fsmap, while one MDS is still up:active on rank 0. ceph tell mds.* status confirms that the MDS for rookfs is: - state: up:active - fs_name: rookfs - whoami: 0 I ran the following commands on the CephFS journal: 1. Journal reset: bash cephfs-journal-tool --rank=rookfs:0 journal reset This completed and indicated a new journal start offset. 2. Journal inspection: bash cephfs-journal-tool --rank=rookfs:0 journal inspect Output : - Bad entry start ptr (...) at certain offsets - Overall journal integrity: DAMAGED - Corrupt regions reported, including a range up to ffffffffffffffff So, even after the reset, cephfs-journal-tool reports the journal as DAMAGED with corrupt regions. Listing the metadata pool shows at least the mds_snaptable object, so the metadata pool is not empty. *Current behaviour* - ceph fs status is sometimes very slow or appears to hang. - Ceph health reports: - “MDSs report slow metadata IOs” - “MDSs report slow requests” - Stale PGs in the cluster - The filesystem rookfs is marked damaged in ceph fs dump, but the MDS is still up:active on rank 0. Any guidance or best practices for handling this kind of journal corruption and damaged filesystem in a Rook/Kubernetes setup would be greatly appreciated, including precautions you would strongly recommend before running the heavy-repair commands. Best regards, Anthony