test Zitat von Eugen Block <eblock@nde.ag>:
Hi,
it's not really clear to me if you had the NVMe drives for the CephFS metadata pool only or if you had DB/WAL for spinners as well on those drives. I assume it was only the pool on them, regarding your statement that there are active+clean PGs. More comments inline...
Zitat von Enzo Van Thorre via ceph-users <ceph-users@ceph.io>:
Hello Ceph users and readers,
I am facing a really unknown situation for me (being kinda new) with a CephFS cluster (Squid 19.2.2, Cephadm) where the data pool is intact (7.2 TB used, 4.9TB stored), inode objects exist in the recovered metadata pool, but when i try to mount the filesystem, its empty.
What initially happened is that I had two groups of drives: nvme’s and spinning. I had separate pools for both, and I eventually deleted the nvme’s pool and then removed the nvmes from the machine. I never thought that they had included metadata for spinning rust’s pool, but here we are.
For the future: 'ceph osd ok-to-stop <ID>' can be helpful if you're not sure if it's okay to stop an OSD. That could have prevented it from happening.
Version: 19.2.2 Squid Metadata pool ID 1 (cephfs.cephfs.meta), Size=1 (No Replicas :(, yes I know I messed up).
yeah...
Data Pool ID 6 (ec_data), erasure coded with k=2 m=1.
that's not great either (k2, m1)
All OSDs are up. 256 PGs are active+clean. there’s then quite a bit that are inconsistent, degraded, undersized, remapped, and other stuff.
what is "other stuff"? A 'ceph health detail' could be helpful.
Here’s what I’ve done since losing the metadata pool due to OSD purges. I tried to rebuild metadata from the data pool by doing: 1. cephfs-data-scan init --force-init 2. cephfs-data-scan scan_extents 3. cephfs-data-scan scan_inodes
ceph df shows 7.2 TiB stored in pool ec_data (which is the spinning rust pool) Inodes do exist. running rados -p cephfs.cephfs.meta ls | grep "100." returns tons of objects (ex. 1000005f1f3.00000000). If i try to mount via ceph-fuse, it shows an empty root. there is no lost+found folder either.
A big problem is that when I try to link the orphaned inodes to lost+found, the command finishes instantly with no output, as if it believes the cluster is clean.
I run: cephfs-data-scan scan_links --filesystem cephfs # and it returns code 0, no output, and runs in <1 second
So my final question is since scan_inodes has populated the metadata pool with objects (100.xxxx), but scan_links ignores them, how do I force a re-scan or just do SOMETHING so scan_links will actually link these files into lost+found?
Unfortunately, I don't have a comment on that right now. But maybe increasing debug level for MDS can help here?
All The Best, Enzo
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io