Dear ceph-users, we are looking for an advice before taking any destructive action on a production Ceph cluster (18.2.4) after a SAN failure. We lost access to a Disk System hosting several OSDs. The bulk of the recovery/backfills did complete, and most PGs are now `active+clean`, but a small set of PGs remains in status `unavailable`. The Disk System might still be recoverable, but not easily or quickly; it may take days, or it may never come back. We are trying to understand the risk of waiting versus marking the missing OSDs as lost. We have not marked any OSDs as lost yet. This is a Ceph Reef cluster, version 18.2.4. Sanitized current status: health: HEALTH_WARN 1 filesystem is degraded 1 MDS reports slow metadata IOs noscrub,nodeep-scrub flags set (temporarily enabled by us to reduce scrub load during recovery/rebalancing) Reduced data availability: 28 PGs inactive, 12 PGs down slow ops still present osd: 86 OSDs total 69 up/in 17 OSDs unavailable because of the SAN issue pgs: 3320 active+clean 16 unknown 11 down 1 down+remapped The problematic PGs are spread across several services/pools: Main CephFS data pool: 4 PGs down Secondary CephFS metadata/data: 1 metadata PG unknown, 4 data PGs unknown RGW/S3 service A: 2 bucket-data PGs down + 1 metadata PG unknown RGW/S3 service B: 5 mostly empty bucket-data PGs down + 1 metadata PG unknown RBD/Kubernetes: 1 PG unknown The services we care about most are the filesystems, especially the main CephFS and the secondary CephFS. For the main CephFS, the metadata pool does not appear to have inactive PGs. The remaining problematic PGs are in the data pool. Of the main CephFS data PGs, only one appears to contain real data; the others appear empty. For the non-empty main CephFS data PG, `pg query` shows roughly: state: down acting/up: all current acting OSDs are up last_update == last_complete empty: 0 incomplete: 0 stat_sum: num_bytes: ~80 GB num_objects: ~131k num_objects_missing: 0 num_objects_unfound: 0 num_objects_degraded: 0 num_objects_misplaced: 0 blocked: peering is blocked due to down OSDs down_osds_we_would_probe: a few OSDs from previous intervals peering_blocked_by: "starting or marking this osd lost may let us proceed" The recovery state for this PG includes past intervals with OSDs that are currently unavailable. The current acting set is up, but Ceph refuses to activate the PG because it still wants to probe down OSDs from the PG history. We tried to list objects from that PG: rados -p <main_cephfs_data_pool> ls --pgid <pgid> but it hangs/times out, so we cannot currently map RADOS objects back to CephFS inode/path names. For the secondary CephFS, the situation is more concerning because one metadata PG is `unknown`. Most `pg query` attempts on unknown PGs return: Error ENOENT: i don't have pgid <pgid> For the RGW/S3 pools, a couple of bucket-data PGs are non-empty and down, while some others appear empty. Some metadata PGs are unknown. RGW can temporarily work after restarting daemons, but eventually gets stuck again, presumably when requests hit unavailable RADOS objects or metadata. We also tested some RBD/Kubernetes volumes. For the Kubernetes RBD pool, only one non-critical PVC seems to hang on `rbd info`; most other images respond. So the main operational concern remains CephFS and RGW. Our questions are: 1. In a case like this, where a CephFS data PG is `down`, not `incomplete`, and shows `num_objects_missing = 0` and `num_objects_unfound = 0`, but peering is blocked by historical down OSDs, what are the likely consequences of marking those OSDs as lost? 2. If we mark the unavailable OSDs as lost, should we expect the main CephFS to: - come back with only some files or file extents unreadable/corrupt, - remain blocked/degraded, - or risk a wider filesystem-level failure? 3. Since the main CephFS metadata pool does not appear to have inactive PGs, is it reasonable to assume that the risk is mostly limited to file data objects rather than the entire namespace? Or is that assumption unsafe? 4. For the secondary CephFS, where a metadata PG is `unknown`, should we treat declaring OSDs lost as much more dangerous? Could that make the whole filesystem unusable rather than only losing some files? 5. Is there any recommended way to map the affected CephFS data PG to file paths while the PG is still down? `rados ls --pgid` hangs, so we currently cannot list objects in the PG. 6. Before declaring the unavailable OSDs lost, are there any additional diagnostic or recovery steps you would recommend to better assess the situation or reduce the risk of data loss? Any advice would be greatly appreciated. Best regards, Nadir and Alessandro