Hi, I don't really have simple answers here, but I'll comment inline. Zitat von Alessandro De Salvo via ceph-users <ceph-users@ceph.io>:

Dear ceph-users,

we are looking for an advice before taking any destructive action on a production Ceph cluster (18.2.4) after a SAN failure.

We lost access to a Disk System hosting several OSDs. The bulk of the recovery/backfills did complete, and most PGs are now `active+clean`, but a small set of PGs remains in status `unavailable`. The Disk System might still be recoverable, but not easily or quickly; it may take days, or it may never come back. We are trying to understand the risk of waiting versus marking the missing OSDs as lost.

We have not marked any OSDs as lost yet.

This is a Ceph Reef cluster, version 18.2.4.

Sanitized current status:

health: HEALTH_WARN 1 filesystem is degraded 1 MDS reports slow metadata IOs noscrub,nodeep-scrub flags set (temporarily enabled by us to reduce scrub load during recovery/rebalancing) Reduced data availability: 28 PGs inactive, 12 PGs down slow ops still present osd: 86 OSDs total 69 up/in 17 OSDs unavailable because of the SAN issue pgs: 3320 active+clean 16 unknown 11 down 1 down+remapped

The problematic PGs are spread across several services/pools:

Main CephFS data pool: 4 PGs down Secondary CephFS metadata/data: 1 metadata PG unknown, 4 data PGs unknown RGW/S3 service A: 2 bucket-data PGs down + 1 metadata PG unknown RGW/S3 service B: 5 mostly empty bucket-data PGs down + 1 metadata PG unknown RBD/Kubernetes: 1 PG unknown

Could you share the following output: ceph osd tree ceph osd pool ls detail ceph health detail

The services we care about most are the filesystems, especially the main CephFS and the secondary CephFS.

For the main CephFS, the metadata pool does not appear to have inactive PGs. The remaining problematic PGs are in the data pool. Of the main CephFS data PGs, only one appears to contain real data; the others appear empty.

I find it very unlikely that only one of the PGs has data. If there are down OSDs, they can't report to the mgr, so the stats are most likely wrong.

For the non-empty main CephFS data PG, `pg query` shows roughly:

state: down acting/up: all current acting OSDs are up last_update == last_complete empty: 0 incomplete: 0 stat_sum: num_bytes: ~80 GB num_objects: ~131k num_objects_missing: 0 num_objects_unfound: 0 num_objects_degraded: 0 num_objects_misplaced: 0 blocked: peering is blocked due to down OSDs down_osds_we_would_probe: a few OSDs from previous intervals peering_blocked_by: "starting or marking this osd lost may let us proceed"

The recovery state for this PG includes past intervals with OSDs that are currently unavailable. The current acting set is up, but Ceph refuses to activate the PG because it still wants to probe down OSDs from the PG history.

We tried to list objects from that PG: rados -p <main_cephfs_data_pool> ls --pgid <pgid>

but it hangs/times out, so we cannot currently map RADOS objects back to CephFS inode/path names.

For the secondary CephFS, the situation is more concerning because one metadata PG is `unknown`. Most `pg query` attempts on unknown PGs return: Error ENOENT: i don't have pgid <pgid>

Same applies here, down OSDs can't report anything.

For the RGW/S3 pools, a couple of bucket-data PGs are non-empty and down, while some others appear empty. Some metadata PGs are unknown. RGW can temporarily work after restarting daemons, but eventually gets stuck again, presumably when requests hit unavailable RADOS objects or metadata.

We also tested some RBD/Kubernetes volumes. For the Kubernetes RBD pool, only one non-critical PVC seems to hang on `rbd info`; most other images respond. So the main operational concern remains CephFS and RGW.

Our questions are: 1. In a case like this, where a CephFS data PG is `down`, not `incomplete`, and shows `num_objects_missing = 0` and `num_objects_unfound = 0`, but peering is blocked by historical down OSDs, what are the likely consequences of marking those OSDs as lost?

Data loss. IIRC, a down OSD means there are no OSDs to query at all (for that PG) while incomplete means that there are not enough OSDs "up" to determine which one has the most up-to-date state.

2. If we mark the unavailable OSDs as lost, should we expect the main CephFS to: - come back with only some files or file extents unreadable/corrupt, - remain blocked/degraded, - or risk a wider filesystem-level failure?

That's difficult to answer. Losing an entire OSD can mean several TB of data lost, so it's unlikely that "only some files or file extents" will be lost. I think it would allow Ceph to recover further, most likely you'd have to scrub the FS afterwards, but at some point you should have a working FS, I think.

3. Since the main CephFS metadata pool does not appear to have inactive PGs, is it reasonable to assume that the risk is mostly limited to file data objects rather than the entire namespace? Or is that assumption unsafe?

I tend to concur, but what do you mean by "does not appear to have inactive PGs"? The 'ceph health detail' output should have a clear answer to that.

4. For the secondary CephFS, where a metadata PG is `unknown`, should we treat declaring OSDs lost as much more dangerous? Could that make the whole filesystem unusable rather than only losing some files?

There are ways to recover the metadata, but be very careful with that! https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/

5. Is there any recommended way to map the affected CephFS data PG to file paths while the PG is still down? `rados ls --pgid` hangs, so we currently cannot list objects in the PG.

I don't think so.

6. Before declaring the unavailable OSDs lost, are there any additional diagnostic or recovery steps you would recommend to better assess the situation or reduce the risk of data loss?

Is there no chance to access the missing OSDs? So one could at least export PGs or something?

Any advice would be greatly appreciated.

Best regards,

    Nadir and Alessandro

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io