Hi, is this an EC pool, maybe k4 m2? If it is, have you tried setting min_size to k for this pool? And if you do that, please don't forget to set it back to k + 1. Do you have the 'ceph pg 133.186 query' output for this PG? What's the health status of the rest of the cluster? I'm not entirely sure in which state a PG has to be to be able to deep-scrub, but this one is in forced_backfill state. Maybe if you cancelled that (ceph pg cancel-force-backfill) a scrub could be scheduled? What's the scrub status of that PG anyway ('ceph pg dump' has that information)? And one more thing, according to this: [NONE,233,768,1202,NONE,423] OSD.233 is not the primary OSD but the secondary. Regards, Eugen Zitat von Michal Strnad via ceph-users <ceph-users@ceph.io>:

Hi.

We have a single unfound object. All OSDs are up, so a simple “put down to up” of an OSD is unlikely to recover it.

Before we notify the object owner and mark the object as lost, I would like to scrub the PG to confirm that only this one object is missing. However, we are unable to force either a normal scrub or a deep scrub on the affected PG (e.g., ceph pg scrub 133.186 or ceph pg deep‑scrub 133.186). We even increased osd_max_scrubs from 1 to 3 on all OSDs (which are participating on PG 133.186):

for i in 328 233 768 1202 84 423; do ceph config set osd.$i osd_max_scrubs 3 done

Despite this, the scrub still does not start. We have set osd_scrub_during_recovery to true. I am attaching the configuration of the primary OSD (osd.233); the other OSDs have identical settings.

WHO MASK LEVEL OPTION VALUE RO global basic device_failure_prediction_mode local osd advanced osd_max_backfills 10 osd.233 advanced osd_max_scrubs 3 osd advanced osd_mclock_override_recovery_settings false osd advanced osd_op_queue wpq * global advanced osd_pool_default_pg_autoscale_mode warn osd advanced osd_repair_during_recovery true osd advanced osd_scrub_auto_repair true osd advanced osd_scrub_during_recovery true osd.233 advanced osd_scrub_load_threshold 10.000000

State of PG:

pg 133.186 is stuck undersized for 6d, current state active+backfill_unfound+undersized+degraded+remapped+forced_backfill, last acting [NONE,233,768,1202,NONE,423]

Our ceph version: 18.2.1 (Reef)

Do you have any ideas on how to get the scrub to run, or on alternative steps we should take to verify that only this single object is lost?

Best regards, Michal