Hi Just a follow up, the issue was solved by running command ceph pg 404.1ff mark_unfound_lost delete - Kai Stian Olstad On 04.01.2023 13:00, Kai Stian Olstad wrote:

Hi

We are running Ceph 16.2.6 deployed with Cephadm.

Around Christmas OSD 245 and 327 had about 20 read error so I set them to out.

Around new year another OSD 313 more or less died since is become so slow that it triggered Linux default I/O-timeout of 30 seconds. In this period the OSD crashed 8 times and was restartet by Systemd and we ended up with

[WRN] OBJECT_UNFOUND: 1/416287126 objects unfound (0.000%) pg 404.1ff has 1 unfound objects [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound pg 404.1ff is active+recovery_unfound+degraded+remapped, acting [208,220,269,175,313,329], 1 unfound [WRN] PG_DEGRADED: Degraded data redundancy: 5/2364745884 objects degraded (0.000%), 1 pg degraded pg 404.1ff is active+recovery_unfound+degraded+remapped, acting [208,220,269,175,313,329], 1 unfound

The pool 404 is "default.rgw.buckets.data" and pool 404 is erasure encoding 4+2.

I have search for a solution but with no luck, what I have tried is

- Restarted all 6 OSD for the PG one by one - Running repair of 404.1ff

Output of following command - ceph -s - ceph health detail - ceph pg ls | grep -e PG -e ^404.1ff - ceph osd pool ls detail | grep 404 - ceph osd tree out - ceph crash ls | grep -e ID -e osd.313 - ceph pg 404.1ff list_unfound - ceph pg 404.1ff

Is appended below, can also be read here https://gitlab.com/-/snippets/2479624 or cloned with "git clone https://gitlab.com/-/snippets/2479624"

Does anyone have any idea on how to resolv the problem? Any help is much appreciated.

- Kai Stian Olstad

:::::::::::::: ceph-s.txt :::::::::::::: ceph -s ------- cluster: id: d13c6b81-51ee-4d22-84e9-456f9307296c health: HEALTH_ERR 1/416287125 objects unfound (0.000%) Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 5/2364745860 objects degraded (0.000%), 1 pg degraded

services: mon: 3 daemons, quorum ceph-mon-1,ceph-mon-2,ceph-mon-3 (age 2M) mgr: ceph-mon-2.mfdanx(active, since 3w), standbys: ceph-mon-1.ptrsea mds: 1/1 daemons up, 1 standby osd: 355 osds: 355 up (since 20h), 352 in (since 2d); 1 remapped pgs rgw: 4 daemons active (4 hosts, 1 zones)

data: volumes: 1/1 healthy pools: 14 pools, 2505 pgs objects: 416.29M objects, 540 TiB usage: 939 TiB used, 2.1 PiB / 3.0 PiB avail pgs: 5/2364745860 objects degraded (0.000%) 137931/2364745860 objects misplaced (0.006%) 1/416287125 objects unfound (0.000%) 2489 active+clean 14 active+clean+scrubbing+deep 1 active+recovery_unfound+degraded+remapped 1 active+clean+scrubbing

io: client: 38 MiB/s rd, 23 MiB/s wr, 2.58k op/s rd, 326 op/s wr

progress: Global Recovery Event (6d) [===========================.] (remaining: 3m)

:::::::::::::: ceph_health_detail.txt :::::::::::::: ceph health detail ------------------ HEALTH_ERR 1/416287126 objects unfound (0.000%); Possible data damage: 1 pg recovery_unfound; Degraded data redundancy: 5/2364745884 objects degraded (0.000%), 1 pg degraded [WRN] OBJECT_UNFOUND: 1/416287126 objects unfound (0.000%) pg 404.1ff has 1 unfound objects [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound pg 404.1ff is active+recovery_unfound+degraded+remapped, acting [208,220,269,175,313,329], 1 unfound [WRN] PG_DEGRADED: Degraded data redundancy: 5/2364745884 objects degraded (0.000%), 1 pg degraded pg 404.1ff is active+recovery_unfound+degraded+remapped, acting [208,220,269,175,313,329], 1 unfound

:::::::::::::: ceph_pg_ls.txt :::::::::::::: ceph pg ls | grep -e PG -e ^404.1ff ----------------------------------- PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP 404.1ff 137912 5 137908 1 282417561722 0 0 5528 active+recovery_unfound+degraded+remapped 19h 141748'724163 141748:3558203 [208,220,269,175,343,329]p208 [208,220,269,175,313,329]p208 2022-12-31T19:27:10.993286+0000 2022-12-31T19:27:10.993286+0000

:::::::::::::: ceph_osd_pool_ls_detail.txt :::::::::::::: ceph osd pool ls detail | grep 404 ---------------------------------- pool 404 'default.rgw.buckets.data' erasure profile ec42-jerasure-blaum_roth-hdd size 6 min_size 5 crush_rule 2 object_hash rjenkins pg_num 2048 pgp_num 2048 autoscale_mode on last_change 124077 lfor 0/52091/108555 flags hashpspool stripe_width 229376 target_size_bytes 1099511627776000 application rgw

:::::::::::::: ceph_osd_tree_out.txt :::::::::::::: ceph osd tree out ----------------- ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 3112.43481 root default -58 192.35847 host ceph-hd-010 245 hdd 12.82390 osd.245 up 0 1.00000 -79 192.35847 host ceph-hd-014 313 hdd 12.82390 osd.313 up 0 1.00000 327 hdd 12.82390 osd.327 up 0 1.00000

:::::::::::::: ceph_crash_ls.txt :::::::::::::: ceph crash ls | grep -e ID -e osd.313 ------------------------------------- ID ENTITY NEW 2022-12-31T01:19:06.859805Z_8e94201c-8817-4c4e-b4dd-67c5b1d3a73a osd.313 2022-12-31T01:19:47.265925Z_551b2de6-d1b0-4cbc-9e62-27ede7a46803 osd.313 2022-12-31T01:20:18.648279Z_64f232ec-2cb5-4dcf-8686-b12987ed7e45 osd.313 2022-12-31T20:28:43.093203Z_9c9a590f-7490-4f74-affe-5bda27b87618 osd.313 2023-01-02T01:34:25.087309Z_b7a4b341-5f79-467b-a4e2-3df5456a2ff7 osd.313 2023-01-02T03:46:32.295053Z_d71b5b62-5974-4b3e-b1c5-cc878d376f06 osd.313 2023-01-02T10:58:58.350473Z_ca87b412-bae5-4873-9997-092587d7c699 osd.313 2023-01-02T11:42:13.433011Z_c0e9ab85-a162-4331-b690-68847c641bd0 osd.313

:::::::::::::: ceph_pg_404.1ff_list_unfound.json :::::::::::::: { "num_missing": 1, "num_unfound": 1, "objects": [ { "oid": { "oid": "1f244892-a2e7-406b-aa62-1b13511333a2.902205.1__multipart_ingest/date=2022-12-30/2022-12-30.json.gz.2~pORMz8G-kyQx8weY8GhNt-pgAmQDufG.684", "key": "", "snapid": -2, "hash": 1069593087, "max": 0, "pool": 404, "namespace": "" }, "need": "140483'722076", "have": "0'0", "flags": "none", "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", "locations": [ "329(5)" ] } ], "state": "NotRecovering", "available_might_have_unfound": true, "might_have_unfound": [], "more": false }

:::::::::::::: ceph_pg_404.1ff.json :::::::::::::: { "snap_trimq": "[]", "snap_trimq_len": 0, "state": "active+recovery_unfound+degraded+remapped", "epoch": 141748, "up": [ 208, 220, 269, 175, 343, 329 ], "acting": [ 208, 220, 269, 175, 313, 329 ], "backfill_targets": [ "343(4)" ], "acting_recovery_backfill": [ "175(3)", "208(0)", "220(1)", "269(2)", "313(4)", "329(5)", "343(4)" ], "info": { "pgid": "404.1ffs0", "last_update": "141747'724161", "last_complete": "140483'722075", "log_tail": "138455'718635", "last_user_version": 724161, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 36041, "epoch_pool_created": 27862, "last_epoch_started": 141660, "last_interval_started": 141659, "last_epoch_clean": 140452, "last_interval_clean": 140451, "last_epoch_split": 108548, "last_epoch_marked_full": 0, "same_up_since": 141659, "same_interval_since": 141659, "same_primary_since": 141615, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "141747'724161", "reported_seq": 3558199, "reported_epoch": 141748, "state": "active+recovery_unfound+degraded+remapped", "last_fresh": "2023-01-04T07:48:25.201886+0000", "last_change": "2023-01-03T12:15:50.155062+0000", "last_active": "2023-01-04T07:48:25.201886+0000", "last_peered": "2023-01-04T07:48:25.201886+0000", "last_clean": "2022-12-31T19:25:42.758665+0000", "last_became_active": "2023-01-03T12:15:49.875672+0000", "last_became_peered": "2023-01-03T12:15:49.875672+0000", "last_unstale": "2023-01-04T07:48:25.201886+0000", "last_undegraded": "2023-01-03T12:15:49.758975+0000", "last_fullsized": "2023-01-04T07:48:25.201886+0000", "mapping_epoch": 141659, "log_start": "138455'718635", "ondisk_log_start": "138455'718635", "created": 36041, "last_epoch_clean": 140452, "parent": "0.0", "parent_split_bits": 10, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "log_size": 5526, "ondisk_log_size": 5526, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 282409304186, "num_objects": 137910, "num_object_clones": 0, "num_object_copies": 827460, "num_objects_missing_on_primary": 1, "num_objects_missing": 1, "num_objects_degraded": 5, "num_objects_misplaced": 137906, "num_objects_unfound": 1, "num_objects_dirty": 137910, "num_whiteouts": 0, "num_read": 732600, "num_read_kb": 181255155, "num_write": 471445, "num_write_kb": 102310260, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 406583, "num_bytes_recovered": 891273764570, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 3 }, "up": [ 208, 220, 269, 175, 343, 329 ], "acting": [ 208, 220, 269, 175, 313, 329 ], "avail_no_missing": [ "329(5)" ], "object_location_counts": [ { "shards": "175(3),208(0),220(1),269(2),313(4),329(5)", "objects": 137909 }, { "shards": "329(5)", "objects": 1 } ], "blocked_by": [], "up_primary": 208, "acting_primary": 208, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 141660, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "175(3)", "pgid": "404.1ffs3", "last_update": "141747'724161", "last_complete": "141668'723872", "log_tail": "138378'718235", "last_user_version": 723769, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 36041, "epoch_pool_created": 27862, "last_epoch_started": 141660, "last_interval_started": 141659, "last_epoch_clean": 140452, "last_interval_clean": 140451, "last_epoch_split": 108548, "last_epoch_marked_full": 0, "same_up_since": 141659, "same_interval_since": 141659, "same_primary_since": 141615, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "141624'723769", "reported_seq": 3550912, "reported_epoch": 141624, "state": "active+recovery_unfound+degraded+remapped", "last_fresh": "2023-01-03T11:57:15.480994+0000", "last_change": "2023-01-03T11:31:07.335605+0000", "last_active": "2023-01-03T11:57:15.480994+0000", "last_peered": "2023-01-03T11:57:15.480994+0000", "last_clean": "2022-12-31T19:25:42.758665+0000", "last_became_active": "2023-01-03T11:31:07.116154+0000", "last_became_peered": "2023-01-03T11:31:07.116154+0000", "last_unstale": "2023-01-03T11:57:15.480994+0000", "last_undegraded": "2023-01-03T11:31:07.099558+0000", "last_fullsized": "2023-01-03T11:57:15.480994+0000", "mapping_epoch": 141659, "log_start": "138378'718235", "ondisk_log_start": "138378'718235", "created": 36041, "last_epoch_clean": 140452, "parent": "0.0", "parent_split_bits": 10, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "log_size": 5534, "ondisk_log_size": 5534, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 281850993538, "num_objects": 137711, "num_object_clones": 0, "num_object_copies": 826266, "num_objects_missing_on_primary": 1, "num_objects_missing": 1, "num_objects_degraded": 5, "num_objects_misplaced": 137707, "num_objects_unfound": 1, "num_objects_dirty": 137711, "num_whiteouts": 0, "num_read": 720741, "num_read_kb": 180044133, "num_write": 469883, "num_write_kb": 101652239, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 406583, "num_bytes_recovered": 891273764570, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 3 }, "up": [ 208, 220, 269, 175, 343, 329 ], "acting": [ 208, 220, 269, 175, 313, 329 ], "avail_no_missing": [ "329(5)" ], "object_location_counts": [ { "shards": "175(3),208(0),220(1),269(2),313(4),329(5)", "objects": 137710 }, { "shards": "329(5)", "objects": 1 } ], "blocked_by": [], "up_primary": 208, "acting_primary": 208, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 141660, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "220(1)", "pgid": "404.1ffs1", "last_update": "141747'724161", "last_complete": "141668'723872", "log_tail": "138378'718235", "last_user_version": 723769, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 36041, "epoch_pool_created": 27862, "last_epoch_started": 141660, "last_interval_started": 141659, "last_epoch_clean": 140452, "last_interval_clean": 140451, "last_epoch_split": 108548, "last_epoch_marked_full": 0, "same_up_since": 141659, "same_interval_since": 141659, "same_primary_since": 141615, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "141624'723769", "reported_seq": 3550912, "reported_epoch": 141624, "state": "active+recovery_unfound+degraded+remapped", "last_fresh": "2023-01-03T11:57:15.480994+0000", "last_change": "2023-01-03T11:31:07.335605+0000", "last_active": "2023-01-03T11:57:15.480994+0000", "last_peered": "2023-01-03T11:57:15.480994+0000", "last_clean": "2022-12-31T19:25:42.758665+0000", "last_became_active": "2023-01-03T11:31:07.116154+0000", "last_became_peered": "2023-01-03T11:31:07.116154+0000", "last_unstale": "2023-01-03T11:57:15.480994+0000", "last_undegraded": "2023-01-03T11:31:07.099558+0000", "last_fullsized": "2023-01-03T11:57:15.480994+0000", "mapping_epoch": 141659, "log_start": "138378'718235", "ondisk_log_start": "138378'718235", "created": 36041, "last_epoch_clean": 140452, "parent": "0.0", "parent_split_bits": 10, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "log_size": 5534, "ondisk_log_size": 5534, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 281850993538, "num_objects": 137711, "num_object_clones": 0, "num_object_copies": 826266, "num_objects_missing_on_primary": 1, "num_objects_missing": 1, "num_objects_degraded": 5, "num_objects_misplaced": 137707, "num_objects_unfound": 1, "num_objects_dirty": 137711, "num_whiteouts": 0, "num_read": 720741, "num_read_kb": 180044133, "num_write": 469883, "num_write_kb": 101652239, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 406583, "num_bytes_recovered": 891273764570, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 3 }, "up": [ 208, 220, 269, 175, 343, 329 ], "acting": [ 208, 220, 269, 175, 313, 329 ], "avail_no_missing": [ "329(5)" ], "object_location_counts": [ { "shards": "175(3),208(0),220(1),269(2),313(4),329(5)", "objects": 137710 }, { "shards": "329(5)", "objects": 1 } ], "blocked_by": [], "up_primary": 208, "acting_primary": 208, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 141660, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "269(2)", "pgid": "404.1ffs2", "last_update": "141747'724161", "last_complete": "141668'723872", "log_tail": "138378'718235", "last_user_version": 723769, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 36041, "epoch_pool_created": 27862, "last_epoch_started": 141660, "last_interval_started": 141659, "last_epoch_clean": 140452, "last_interval_clean": 140451, "last_epoch_split": 108548, "last_epoch_marked_full": 0, "same_up_since": 141659, "same_interval_since": 141659, "same_primary_since": 141615, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "141624'723769", "reported_seq": 3550912, "reported_epoch": 141624, "state": "active+recovery_unfound+degraded+remapped", "last_fresh": "2023-01-03T11:57:15.480994+0000", "last_change": "2023-01-03T11:31:07.335605+0000", "last_active": "2023-01-03T11:57:15.480994+0000", "last_peered": "2023-01-03T11:57:15.480994+0000", "last_clean": "2022-12-31T19:25:42.758665+0000", "last_became_active": "2023-01-03T11:31:07.116154+0000", "last_became_peered": "2023-01-03T11:31:07.116154+0000", "last_unstale": "2023-01-03T11:57:15.480994+0000", "last_undegraded": "2023-01-03T11:31:07.099558+0000", "last_fullsized": "2023-01-03T11:57:15.480994+0000", "mapping_epoch": 141659, "log_start": "138378'718235", "ondisk_log_start": "138378'718235", "created": 36041, "last_epoch_clean": 140452, "parent": "0.0", "parent_split_bits": 10, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "log_size": 5534, "ondisk_log_size": 5534, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 281850993538, "num_objects": 137711, "num_object_clones": 0, "num_object_copies": 826266, "num_objects_missing_on_primary": 1, "num_objects_missing": 1, "num_objects_degraded": 5, "num_objects_misplaced": 137707, "num_objects_unfound": 1, "num_objects_dirty": 137711, "num_whiteouts": 0, "num_read": 720741, "num_read_kb": 180044133, "num_write": 469883, "num_write_kb": 101652239, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 406583, "num_bytes_recovered": 891273764570, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 3 }, "up": [ 208, 220, 269, 175, 343, 329 ], "acting": [ 208, 220, 269, 175, 313, 329 ], "avail_no_missing": [ "329(5)" ], "object_location_counts": [ { "shards": "175(3),208(0),220(1),269(2),313(4),329(5)", "objects": 137710 }, { "shards": "329(5)", "objects": 1 } ], "blocked_by": [], "up_primary": 208, "acting_primary": 208, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 141660, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "313(4)", "pgid": "404.1ffs4", "last_update": "141747'724161", "last_complete": "141668'723872", "log_tail": "138228'717435", "last_user_version": 723769, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 36041, "epoch_pool_created": 27862, "last_epoch_started": 141660, "last_interval_started": 141659, "last_epoch_clean": 140452, "last_interval_clean": 140451, "last_epoch_split": 108548, "last_epoch_marked_full": 0, "same_up_since": 141659, "same_interval_since": 141659, "same_primary_since": 141615, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "141624'723769", "reported_seq": 3551314, "reported_epoch": 141659, "state": "active+recovery_unfound+degraded+remapped", "last_fresh": "2023-01-03T11:57:15.480994+0000", "last_change": "2023-01-03T11:31:07.335605+0000", "last_active": "2023-01-03T11:57:15.480994+0000", "last_peered": "2023-01-03T11:57:15.480994+0000", "last_clean": "2022-12-31T19:25:42.758665+0000", "last_became_active": "2023-01-03T11:31:07.116154+0000", "last_became_peered": "2023-01-03T11:31:07.116154+0000", "last_unstale": "2023-01-03T11:57:15.480994+0000", "last_undegraded": "2023-01-03T11:31:07.099558+0000", "last_fullsized": "2023-01-03T11:57:15.480994+0000", "mapping_epoch": 141659, "log_start": "138378'718235", "ondisk_log_start": "138378'718235", "created": 36041, "last_epoch_clean": 140452, "parent": "0.0", "parent_split_bits": 10, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "log_size": 5534, "ondisk_log_size": 5534, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 281850993538, "num_objects": 137711, "num_object_clones": 0, "num_object_copies": 826266, "num_objects_missing_on_primary": 1, "num_objects_missing": 1, "num_objects_degraded": 5, "num_objects_misplaced": 137707, "num_objects_unfound": 1, "num_objects_dirty": 137711, "num_whiteouts": 0, "num_read": 720741, "num_read_kb": 180044133, "num_write": 469883, "num_write_kb": 101652239, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 406583, "num_bytes_recovered": 891273764570, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 3 }, "up": [ 208, 220, 269, 175, 343, 329 ], "acting": [ 208, 220, 269, 175, 313, 329 ], "avail_no_missing": [ "329(5)" ], "object_location_counts": [ { "shards": "175(3),208(0),220(1),269(2),313(4),329(5)", "objects": 137710 }, { "shards": "329(5)", "objects": 1 } ], "blocked_by": [], "up_primary": 208, "acting_primary": 208, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 141660, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "329(5)", "pgid": "404.1ffs5", "last_update": "141747'724161", "last_complete": "141668'723872", "log_tail": "138378'718235", "last_user_version": 723769, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 36041, "epoch_pool_created": 27862, "last_epoch_started": 141660, "last_interval_started": 141659, "last_epoch_clean": 140452, "last_interval_clean": 140451, "last_epoch_split": 108548, "last_epoch_marked_full": 0, "same_up_since": 141659, "same_interval_since": 141659, "same_primary_since": 141615, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "141624'723769", "reported_seq": 3550912, "reported_epoch": 141624, "state": "active+recovery_unfound+degraded+remapped", "last_fresh": "2023-01-03T11:57:15.480994+0000", "last_change": "2023-01-03T11:31:07.335605+0000", "last_active": "2023-01-03T11:57:15.480994+0000", "last_peered": "2023-01-03T11:57:15.480994+0000", "last_clean": "2022-12-31T19:25:42.758665+0000", "last_became_active": "2023-01-03T11:31:07.116154+0000", "last_became_peered": "2023-01-03T11:31:07.116154+0000", "last_unstale": "2023-01-03T11:57:15.480994+0000", "last_undegraded": "2023-01-03T11:31:07.099558+0000", "last_fullsized": "2023-01-03T11:57:15.480994+0000", "mapping_epoch": 141659, "log_start": "138378'718235", "ondisk_log_start": "138378'718235", "created": 36041, "last_epoch_clean": 140452, "parent": "0.0", "parent_split_bits": 10, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "log_size": 5534, "ondisk_log_size": 5534, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 281850993538, "num_objects": 137711, "num_object_clones": 0, "num_object_copies": 826266, "num_objects_missing_on_primary": 1, "num_objects_missing": 0, "num_objects_degraded": 5, "num_objects_misplaced": 137707, "num_objects_unfound": 1, "num_objects_dirty": 137711, "num_whiteouts": 0, "num_read": 720741, "num_read_kb": 180044133, "num_write": 469883, "num_write_kb": 101652239, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 406583, "num_bytes_recovered": 891273764570, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 3 }, "up": [ 208, 220, 269, 175, 343, 329 ], "acting": [ 208, 220, 269, 175, 313, 329 ], "avail_no_missing": [ "329(5)" ], "object_location_counts": [ { "shards": "175(3),208(0),220(1),269(2),313(4),329(5)", "objects": 137710 }, { "shards": "329(5)", "objects": 1 } ], "blocked_by": [], "up_primary": 208, "acting_primary": 208, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 141660, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "343(4)", "pgid": "404.1ffs4", "last_update": "141747'724161", "last_complete": "141668'723872", "log_tail": "138378'718235", "last_user_version": 723769, "last_backfill": "404:ff80001b:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-09T12%3a41%3a05,478440857+00%3a00.2~XwOh1_IhVzghlM-zi7DfwoUIZl5UxVt.1:head", "purged_snaps": [], "history": { "epoch_created": 36041, "epoch_pool_created": 27862, "last_epoch_started": 141660, "last_interval_started": 141659, "last_epoch_clean": 140452, "last_interval_clean": 140451, "last_epoch_split": 108548, "last_epoch_marked_full": 0, "same_up_since": 141659, "same_interval_since": 141659, "same_primary_since": 141615, "last_scrub": "140525'722168", "last_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_deep_scrub": "140525'722168", "last_deep_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "last_clean_scrub_stamp": "2022-12-31T19:27:10.993286+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": 0, "reported_epoch": 0, "state": "unknown", "last_fresh": "0.000000", "last_change": "0.000000", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "0.000000", "last_undegraded": "0.000000", "last_fullsized": "0.000000", "mapping_epoch": 141659, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 0, "last_epoch_clean": 0, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "log_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 9404416, "num_objects": 3, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 137907, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 208, 220, 269, 175, 343, 329 ], "acting": [ 208, 220, 269, 175, 313, 329 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 208, "acting_primary": 208, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 1, "last_epoch_started": 141660, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2023-01-03T12:15:49.758685+0000", "might_have_unfound": [ { "osd": "175(3)", "status": "already probed" }, { "osd": "220(1)", "status": "already probed" }, { "osd": "269(2)", "status": "already probed" }, { "osd": "313(4)", "status": "already probed" }, { "osd": "329(5)", "status": "already probed" }, { "osd": "343(4)", "status": "already probed" } ], "recovery_progress": { "backfill_targets": [ "343(4)" ], "waiting_on_backfill": [], "last_backfill_started": "404:ff80001b:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-09T12%3a41%3a05,478440857+00%3a00.2~XwOh1_IhVzghlM-zi7DfwoUIZl5UxVt.1:head", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "recovery_ops": [], "read_ops": [] } } }, { "name": "Started", "enter_time": "2023-01-03T12:15:48.766366+0000" } ], "scrubber": { "epoch_start": "0", "active": false }, "agent_state": {} } _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io