Dear All, Due to a mistake in my "rolling restart" script, one of our ceph clusters now has a number of unfound objects: There is an 8+2 erasure encoded data pool, 3x replicated metadata pool, all data is stored as cephfs. root@ceph7 ceph-archive]# ceph health HEALTH_ERR 24/420880027 objects unfound (0.000%); Possible data damage: 14 pgs recovery_unfound; Degraded data redundancy: 64/4204261148 objects degraded (0.000%), 14 pgs degraded "ceph health detail" gives me a handle on which pgs are affected. e.g: pg 5.f2f has 2 unfound objects pg 5.5c9 has 2 unfound objects pg 5.4c1 has 1 unfound objects and so on... plus more entries of this type: pg 5.6d is active+recovery_unfound+degraded, acting [295,104,57,442,240,338,219,33,150,382], 1 unfound pg 5.3fa is active+recovery_unfound+degraded, acting [343,147,21,131,315,63,214,365,264,437], 2 unfound pg 5.41d is active+recovery_unfound+degraded, acting [20,104,190,377,52,141,418,358,240,289], 1 unfound Digging deeper into one of the bad pg, we see the oid for the two unfound objects: root@ceph7 ceph-archive]# ceph pg 5.f2f list_unfound { "num_missing": 4, "num_unfound": 2, "objects": [ { "oid": { "oid": "1000ba25e49.00000207", "key": "", "snapid": -2, "hash": 854007599, "max": 0, "pool": 5, "namespace": "" }, "need": "22541'3088478", "have": "0'0", "flags": "none", "locations": [ "189(8)", "263(9)" ] }, { "oid": { "oid": "1000bb25a5b.00000091", "key": "", "snapid": -2, "hash": 3637976879, "max": 0, "pool": 5, "namespace": "" }, "need": "22541'3088476", "have": "0'0", "flags": "none", "locations": [ "189(8)", "263(9)" ] } ], "more": false } While it would be nice to recover the data, this cluster is only used for storing backups. As all OSD are up and running, presumably the data blocks are permanently lost? If it's hard / impossible to recover the data, presumably we should now consider using "ceph pg 5.f2f mark_unfound_lost delete" on each affected pg? Finally, can we use the oid to identify the affected files? best regards, Jake -- Jake Grimmett MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK.
This might be related to recent problems with OSDs not being queried for unfound objects properly in some cases (which I think was fixed in master?) Anyways: run ceph pg <pg> query on the affected PGs, check for "might have unfound" and try restarting the OSDs mentioned there. Probably also sufficient to just run "ceph osd down" on the primaries on the affected PGs to get them to re-check. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Feb 3, 2020 at 4:27 PM Jake Grimmett <jog@mrc-lmb.cam.ac.uk> wrote:
Dear All,
Due to a mistake in my "rolling restart" script, one of our ceph clusters now has a number of unfound objects:
There is an 8+2 erasure encoded data pool, 3x replicated metadata pool, all data is stored as cephfs.
root@ceph7 ceph-archive]# ceph health HEALTH_ERR 24/420880027 objects unfound (0.000%); Possible data damage: 14 pgs recovery_unfound; Degraded data redundancy: 64/4204261148 objects degraded (0.000%), 14 pgs degraded
"ceph health detail" gives me a handle on which pgs are affected. e.g: pg 5.f2f has 2 unfound objects pg 5.5c9 has 2 unfound objects pg 5.4c1 has 1 unfound objects and so on...
plus more entries of this type: pg 5.6d is active+recovery_unfound+degraded, acting [295,104,57,442,240,338,219,33,150,382], 1 unfound pg 5.3fa is active+recovery_unfound+degraded, acting [343,147,21,131,315,63,214,365,264,437], 2 unfound pg 5.41d is active+recovery_unfound+degraded, acting [20,104,190,377,52,141,418,358,240,289], 1 unfound
Digging deeper into one of the bad pg, we see the oid for the two unfound objects:
root@ceph7 ceph-archive]# ceph pg 5.f2f list_unfound { "num_missing": 4, "num_unfound": 2, "objects": [ { "oid": { "oid": "1000ba25e49.00000207", "key": "", "snapid": -2, "hash": 854007599, "max": 0, "pool": 5, "namespace": "" }, "need": "22541'3088478", "have": "0'0", "flags": "none", "locations": [ "189(8)", "263(9)" ] }, { "oid": { "oid": "1000bb25a5b.00000091", "key": "", "snapid": -2, "hash": 3637976879, "max": 0, "pool": 5, "namespace": "" }, "need": "22541'3088476", "have": "0'0", "flags": "none", "locations": [ "189(8)", "263(9)" ] } ], "more": false }
While it would be nice to recover the data, this cluster is only used for storing backups.
As all OSD are up and running, presumably the data blocks are permanently lost?
If it's hard / impossible to recover the data, presumably we should now consider using "ceph pg 5.f2f mark_unfound_lost delete" on each affected pg?
Finally, can we use the oid to identify the affected files?
best regards,
Jake
-- Jake Grimmett MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Paul, Many thanks for your helpful suggestions. Yes, we have 13 pgs with "might_have_unfound" entries. (also 1 pgs without "might_have_unfound" stuck in active+recovery_unfound+degraded+repair state) Taking one pg with unfound objects: [root@ceph1 ~]# ceph health detail | grep 5.5c9 pg 5.5c9 has 2 unfound objects pg 5.5c9 is active+recovery_unfound+degraded, acting [347,442,381,215,91,260,31,94,178,302], 2 unfound pg 5.5c9 is active+recovery_unfound+degraded, acting [347,442,381,215,91,260,31,94,178,302], 2 unfound pg 5.5c9 not deep-scrubbed since 2020-01-16 08:05:43.119336 pg 5.5c9 not scrubbed since 2020-01-16 08:05:43.119336 Checking the state: [root@ceph1 ~]# ceph pg 5.5c9 query | jq .recovery_state [ { "name": "Started/Primary/Active", "enter_time": "2020-02-03 09:57:30.982038", "might_have_unfound": [ { "osd": "31(6)", "status": "already probed" }, { "osd": "91(4)", "status": "already probed" }, { "osd": "94(7)", "status": "already probed" }, { "osd": "178(8)", "status": "already probed" }, { "osd": "215(3)", "status": "already probed" }, { "osd": "260(5)", "status": "already probed" }, { "osd": "302(9)", "status": "already probed" }, { "osd": "381(2)", "status": "already probed" }, { "osd": "442(1)", "status": "already probed" } ], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "recovery_ops": [], "read_ops": [] } }, "scrub": { "scrubber.epoch_start": "0", "scrubber.active": false, "scrubber.state": "INACTIVE", "scrubber.start": "MIN", "scrubber.end": "MIN", "scrubber.max_end": "MIN", "scrubber.subset_last_update": "0'0", "scrubber.deep": false, "scrubber.waiting_on_whom": [] } }, { "name": "Started", "enter_time": "2020-02-03 09:57:29.788310" } ] ----------------------------------------------------- Taking your advice, I restart the primary osd for this pg: [root@ceph1 ~]# ceph osd down 347 This doesn't change the output of "ceph pg 5.5c9 query", apart from updating the Started time, and ceph health still shows unfound objects. To fix this, do we need to issue a scrub (or deep scrub) so that the objects can be found? Just in case, I've issued a manual scrub: [root@ceph1 ~]# ceph pg scrub 5.5c9 instructing pg 5.5c9s0 on osd.347 to scrub The cluster is currently busy deleting snapshots, so it may take a while before the scrub starts. best regards, Jake On 2/3/20 6:31 PM, Paul Emmerich wrote:
This might be related to recent problems with OSDs not being queried for unfound objects properly in some cases (which I think was fixed in master?)
Anyways: run ceph pg <pg> query on the affected PGs, check for "might have unfound" and try restarting the OSDs mentioned there. Probably also sufficient to just run "ceph osd down" on the primaries on the affected PGs to get them to re-check.
Paul
-- Jake Grimmett MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK.
participants (2)
-
Jake Grimmett
-
Paul Emmerich