Hi
Just a follow up, the issue was solved by running command
ceph pg 404.1ff mark_unfound_lost delete
-
Kai Stian Olstad
On 04.01.2023 13:00, Kai Stian Olstad wrote:
> Hi
>
> We are running Ceph 16.2.6 deployed with Cephadm.
>
> Around Christmas OSD 245 and 327 had about 20 read error so I set them
> to out.
>
> Around new year another OSD 313 more or less died since is become so
> slow that it triggered Linux default I/O-timeout of 30 seconds.
> In this period the OSD crashed 8 times and was restartet by Systemd
> and we ended up with
>
> [WRN] OBJECT_UNFOUND: 1/416287126 objects unfound (0.000%)
> pg 404.1ff has 1 unfound objects
> [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound
> pg 404.1ff is active+recovery_unfound+degraded+remapped, acting
> [208,220,269,175,313,329], 1 unfound
> [WRN] PG_DEGRADED: Degraded data redundancy: 5/2364745884 objects
> degraded (0.000%), 1 pg degraded
> pg 404.1ff is active+recovery_unfound+degraded+remapped, acting
> [208,220,269,175,313,329], 1 unfound
>
> The pool 404 is "default.rgw.buckets.data" and pool 404 is erasure
> encoding 4+2.
>
> I have search for a solution but with no luck, what I have tried is
>
> - Restarted all 6 OSD for the PG one by one
> - Running repair of 404.1ff
>
> Output of following command
> - ceph -s
> - ceph health detail
> - ceph pg ls | grep -e PG -e ^404.1ff
> - ceph osd pool ls detail | grep 404
> - ceph osd tree out
> - ceph crash ls | grep -e ID -e osd.313
> - ceph pg 404.1ff list_unfound
> - ceph pg 404.1ff
>
> Is appended below, can also be read here
>
https://gitlab.com/-/snippets/2479624
> or cloned with "git clone
https://gitlab.com/-/snippets/2479624"
>
> Does anyone have any idea on how to resolv the problem?
> Any help is much appreciated.
>
> -
> Kai Stian Olstad
>
>
>
> ::::::::::::::
> ceph-s.txt
> ::::::::::::::
> ceph -s
> -------
> cluster:
> id: d13c6b81-51ee-4d22-84e9-456f9307296c
> health: HEALTH_ERR
> 1/416287125 objects unfound (0.000%)
> Possible data damage: 1 pg recovery_unfound
> Degraded data redundancy: 5/2364745860 objects degraded
> (0.000%), 1 pg degraded
>
> services:
> mon: 3 daemons, quorum ceph-mon-1,ceph-mon-2,ceph-mon-3 (age 2M)
> mgr: ceph-mon-2.mfdanx(active, since 3w), standbys:
> ceph-mon-1.ptrsea
> mds: 1/1 daemons up, 1 standby
> osd: 355 osds: 355 up (since 20h), 352 in (since 2d); 1 remapped
> pgs
> rgw: 4 daemons active (4 hosts, 1 zones)
>
> data:
> volumes: 1/1 healthy
> pools: 14 pools, 2505 pgs
> objects: 416.29M objects, 540 TiB
> usage: 939 TiB used, 2.1 PiB / 3.0 PiB avail
> pgs: 5/2364745860 objects degraded (0.000%)
> 137931/2364745860 objects misplaced (0.006%)
> 1/416287125 objects unfound (0.000%)
> 2489 active+clean
> 14 active+clean+scrubbing+deep
> 1 active+recovery_unfound+degraded+remapped
> 1 active+clean+scrubbing
>
> io:
> client: 38 MiB/s rd, 23 MiB/s wr, 2.58k op/s rd, 326 op/s wr
>
> progress:
> Global Recovery Event (6d)
> [===========================.] (remaining: 3m)
>
>
> ::::::::::::::
> ceph_health_detail.txt
> ::::::::::::::
> ceph health detail
> ------------------
> HEALTH_ERR 1/416287126 objects unfound (0.000%); Possible data damage:
> 1 pg recovery_unfound; Degraded data redundancy: 5/2364745884 objects
> degraded (0.000%), 1 pg degraded
> [WRN] OBJECT_UNFOUND: 1/416287126 objects unfound (0.000%)
> pg 404.1ff has 1 unfound objects
> [ERR] PG_DAMAGED: Possible data damage: 1 pg recovery_unfound
> pg 404.1ff is active+recovery_unfound+degraded+remapped, acting
> [208,220,269,175,313,329], 1 unfound
> [WRN] PG_DEGRADED: Degraded data redundancy: 5/2364745884 objects
> degraded (0.000%), 1 pg degraded
> pg 404.1ff is active+recovery_unfound+degraded+remapped, acting
> [208,220,269,175,313,329], 1 unfound
>
>
> ::::::::::::::
> ceph_pg_ls.txt
> ::::::::::::::
> ceph pg ls | grep -e PG -e ^404.1ff
> -----------------------------------
> PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES
> OMAP_BYTES* OMAP_KEYS* LOG STATE
> SINCE VERSION REPORTED UP
> ACTING SCRUB_STAMP
> DEEP_SCRUB_STAMP
> 404.1ff 137912 5 137908 1 282417561722
> 0 0 5528 active+recovery_unfound+degraded+remapped
> 19h 141748'724163 141748:3558203 [208,220,269,175,343,329]p208
> [208,220,269,175,313,329]p208 2022-12-31T19:27:10.993286+0000
> 2022-12-31T19:27:10.993286+0000
>
>
> ::::::::::::::
> ceph_osd_pool_ls_detail.txt
> ::::::::::::::
> ceph osd pool ls detail | grep 404
> ----------------------------------
> pool 404 'default.rgw.buckets.data' erasure profile
> ec42-jerasure-blaum_roth-hdd size 6 min_size 5 crush_rule 2
> object_hash rjenkins pg_num 2048 pgp_num 2048 autoscale_mode on
> last_change 124077 lfor 0/52091/108555 flags hashpspool stripe_width
> 229376 target_size_bytes 1099511627776000 application rgw
>
>
> ::::::::::::::
> ceph_osd_tree_out.txt
> ::::::::::::::
> ceph osd tree out
> -----------------
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> -1 3112.43481 root default
> -58 192.35847 host ceph-hd-010
> 245 hdd 12.82390 osd.245 up 0 1.00000
> -79 192.35847 host ceph-hd-014
> 313 hdd 12.82390 osd.313 up 0 1.00000
> 327 hdd 12.82390 osd.327 up 0 1.00000
>
>
> ::::::::::::::
> ceph_crash_ls.txt
> ::::::::::::::
> ceph crash ls | grep -e ID -e osd.313
> -------------------------------------
> ID
> ENTITY NEW
> 2022-12-31T01:19:06.859805Z_8e94201c-8817-4c4e-b4dd-67c5b1d3a73a
> osd.313
> 2022-12-31T01:19:47.265925Z_551b2de6-d1b0-4cbc-9e62-27ede7a46803
> osd.313
> 2022-12-31T01:20:18.648279Z_64f232ec-2cb5-4dcf-8686-b12987ed7e45
> osd.313
> 2022-12-31T20:28:43.093203Z_9c9a590f-7490-4f74-affe-5bda27b87618
> osd.313
> 2023-01-02T01:34:25.087309Z_b7a4b341-5f79-467b-a4e2-3df5456a2ff7
> osd.313
> 2023-01-02T03:46:32.295053Z_d71b5b62-5974-4b3e-b1c5-cc878d376f06
> osd.313
> 2023-01-02T10:58:58.350473Z_ca87b412-bae5-4873-9997-092587d7c699
> osd.313
> 2023-01-02T11:42:13.433011Z_c0e9ab85-a162-4331-b690-68847c641bd0
> osd.313
>
>
> ::::::::::::::
> ceph_pg_404.1ff_list_unfound.json
> ::::::::::::::
> {
> "num_missing": 1,
> "num_unfound": 1,
> "objects": [
> {
> "oid": {
> "oid":
>
"1f244892-a2e7-406b-aa62-1b13511333a2.902205.1__multipart_ingest/date=2022-12-30/2022-12-30.json.gz.2~pORMz8G-kyQx8weY8GhNt-pgAmQDufG.684",
> "key": "",
> "snapid": -2,
> "hash": 1069593087,
> "max": 0,
> "pool": 404,
> "namespace": ""
> },
> "need": "140483'722076",
> "have": "0'0",
> "flags": "none",
> "clean_regions": "clean_offsets: [], clean_omap: 0,
> new_object: 1",
> "locations": [
> "329(5)"
> ]
> }
> ],
> "state": "NotRecovering",
> "available_might_have_unfound": true,
> "might_have_unfound": [],
> "more": false
> }
>
>
> ::::::::::::::
> ceph_pg_404.1ff.json
> ::::::::::::::
> {
> "snap_trimq": "[]",
> "snap_trimq_len": 0,
> "state": "active+recovery_unfound+degraded+remapped",
> "epoch": 141748,
> "up": [
> 208,
> 220,
> 269,
> 175,
> 343,
> 329
> ],
> "acting": [
> 208,
> 220,
> 269,
> 175,
> 313,
> 329
> ],
> "backfill_targets": [
> "343(4)"
> ],
> "acting_recovery_backfill": [
> "175(3)",
> "208(0)",
> "220(1)",
> "269(2)",
> "313(4)",
> "329(5)",
> "343(4)"
> ],
> "info": {
> "pgid": "404.1ffs0",
> "last_update": "141747'724161",
> "last_complete": "140483'722075",
> "log_tail": "138455'718635",
> "last_user_version": 724161,
> "last_backfill": "MAX",
> "purged_snaps": [],
> "history": {
> "epoch_created": 36041,
> "epoch_pool_created": 27862,
> "last_epoch_started": 141660,
> "last_interval_started": 141659,
> "last_epoch_clean": 140452,
> "last_interval_clean": 140451,
> "last_epoch_split": 108548,
> "last_epoch_marked_full": 0,
> "same_up_since": 141659,
> "same_interval_since": 141659,
> "same_primary_since": 141615,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "prior_readable_until_ub": 0
> },
> "stats": {
> "version": "141747'724161",
> "reported_seq": 3558199,
> "reported_epoch": 141748,
> "state":
"active+recovery_unfound+degraded+remapped",
> "last_fresh": "2023-01-04T07:48:25.201886+0000",
> "last_change": "2023-01-03T12:15:50.155062+0000",
> "last_active": "2023-01-04T07:48:25.201886+0000",
> "last_peered": "2023-01-04T07:48:25.201886+0000",
> "last_clean": "2022-12-31T19:25:42.758665+0000",
> "last_became_active":
"2023-01-03T12:15:49.875672+0000",
> "last_became_peered":
"2023-01-03T12:15:49.875672+0000",
> "last_unstale": "2023-01-04T07:48:25.201886+0000",
> "last_undegraded":
"2023-01-03T12:15:49.758975+0000",
> "last_fullsized": "2023-01-04T07:48:25.201886+0000",
> "mapping_epoch": 141659,
> "log_start": "138455'718635",
> "ondisk_log_start": "138455'718635",
> "created": 36041,
> "last_epoch_clean": 140452,
> "parent": "0.0",
> "parent_split_bits": 10,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "log_size": 5526,
> "ondisk_log_size": 5526,
> "stats_invalid": false,
> "dirty_stats_invalid": false,
> "omap_stats_invalid": false,
> "hitset_stats_invalid": false,
> "hitset_bytes_stats_invalid": false,
> "pin_stats_invalid": false,
> "manifest_stats_invalid": false,
> "snaptrimq_len": 0,
> "stat_sum": {
> "num_bytes": 282409304186,
> "num_objects": 137910,
> "num_object_clones": 0,
> "num_object_copies": 827460,
> "num_objects_missing_on_primary": 1,
> "num_objects_missing": 1,
> "num_objects_degraded": 5,
> "num_objects_misplaced": 137906,
> "num_objects_unfound": 1,
> "num_objects_dirty": 137910,
> "num_whiteouts": 0,
> "num_read": 732600,
> "num_read_kb": 181255155,
> "num_write": 471445,
> "num_write_kb": 102310260,
> "num_scrub_errors": 0,
> "num_shallow_scrub_errors": 0,
> "num_deep_scrub_errors": 0,
> "num_objects_recovered": 406583,
> "num_bytes_recovered": 891273764570,
> "num_keys_recovered": 0,
> "num_objects_omap": 0,
> "num_objects_hit_set_archive": 0,
> "num_bytes_hit_set_archive": 0,
> "num_flush": 0,
> "num_flush_kb": 0,
> "num_evict": 0,
> "num_evict_kb": 0,
> "num_promote": 0,
> "num_flush_mode_high": 0,
> "num_flush_mode_low": 0,
> "num_evict_mode_some": 0,
> "num_evict_mode_full": 0,
> "num_objects_pinned": 0,
> "num_legacy_snapsets": 0,
> "num_large_omap_objects": 0,
> "num_objects_manifest": 0,
> "num_omap_bytes": 0,
> "num_omap_keys": 0,
> "num_objects_repaired": 3
> },
> "up": [
> 208,
> 220,
> 269,
> 175,
> 343,
> 329
> ],
> "acting": [
> 208,
> 220,
> 269,
> 175,
> 313,
> 329
> ],
> "avail_no_missing": [
> "329(5)"
> ],
> "object_location_counts": [
> {
> "shards":
> "175(3),208(0),220(1),269(2),313(4),329(5)",
> "objects": 137909
> },
> {
> "shards": "329(5)",
> "objects": 1
> }
> ],
> "blocked_by": [],
> "up_primary": 208,
> "acting_primary": 208,
> "purged_snaps": []
> },
> "empty": 0,
> "dne": 0,
> "incomplete": 0,
> "last_epoch_started": 141660,
> "hit_set_history": {
> "current_last_update": "0'0",
> "history": []
> }
> },
> "peer_info": [
> {
> "peer": "175(3)",
> "pgid": "404.1ffs3",
> "last_update": "141747'724161",
> "last_complete": "141668'723872",
> "log_tail": "138378'718235",
> "last_user_version": 723769,
> "last_backfill": "MAX",
> "purged_snaps": [],
> "history": {
> "epoch_created": 36041,
> "epoch_pool_created": 27862,
> "last_epoch_started": 141660,
> "last_interval_started": 141659,
> "last_epoch_clean": 140452,
> "last_interval_clean": 140451,
> "last_epoch_split": 108548,
> "last_epoch_marked_full": 0,
> "same_up_since": 141659,
> "same_interval_since": 141659,
> "same_primary_since": 141615,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "prior_readable_until_ub": 0
> },
> "stats": {
> "version": "141624'723769",
> "reported_seq": 3550912,
> "reported_epoch": 141624,
> "state":
"active+recovery_unfound+degraded+remapped",
> "last_fresh": "2023-01-03T11:57:15.480994+0000",
> "last_change":
"2023-01-03T11:31:07.335605+0000",
> "last_active":
"2023-01-03T11:57:15.480994+0000",
> "last_peered":
"2023-01-03T11:57:15.480994+0000",
> "last_clean": "2022-12-31T19:25:42.758665+0000",
> "last_became_active":
> "2023-01-03T11:31:07.116154+0000",
> "last_became_peered":
> "2023-01-03T11:31:07.116154+0000",
> "last_unstale":
"2023-01-03T11:57:15.480994+0000",
> "last_undegraded":
"2023-01-03T11:31:07.099558+0000",
> "last_fullsized":
"2023-01-03T11:57:15.480994+0000",
> "mapping_epoch": 141659,
> "log_start": "138378'718235",
> "ondisk_log_start": "138378'718235",
> "created": 36041,
> "last_epoch_clean": 140452,
> "parent": "0.0",
> "parent_split_bits": 10,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "log_size": 5534,
> "ondisk_log_size": 5534,
> "stats_invalid": false,
> "dirty_stats_invalid": false,
> "omap_stats_invalid": false,
> "hitset_stats_invalid": false,
> "hitset_bytes_stats_invalid": false,
> "pin_stats_invalid": false,
> "manifest_stats_invalid": false,
> "snaptrimq_len": 0,
> "stat_sum": {
> "num_bytes": 281850993538,
> "num_objects": 137711,
> "num_object_clones": 0,
> "num_object_copies": 826266,
> "num_objects_missing_on_primary": 1,
> "num_objects_missing": 1,
> "num_objects_degraded": 5,
> "num_objects_misplaced": 137707,
> "num_objects_unfound": 1,
> "num_objects_dirty": 137711,
> "num_whiteouts": 0,
> "num_read": 720741,
> "num_read_kb": 180044133,
> "num_write": 469883,
> "num_write_kb": 101652239,
> "num_scrub_errors": 0,
> "num_shallow_scrub_errors": 0,
> "num_deep_scrub_errors": 0,
> "num_objects_recovered": 406583,
> "num_bytes_recovered": 891273764570,
> "num_keys_recovered": 0,
> "num_objects_omap": 0,
> "num_objects_hit_set_archive": 0,
> "num_bytes_hit_set_archive": 0,
> "num_flush": 0,
> "num_flush_kb": 0,
> "num_evict": 0,
> "num_evict_kb": 0,
> "num_promote": 0,
> "num_flush_mode_high": 0,
> "num_flush_mode_low": 0,
> "num_evict_mode_some": 0,
> "num_evict_mode_full": 0,
> "num_objects_pinned": 0,
> "num_legacy_snapsets": 0,
> "num_large_omap_objects": 0,
> "num_objects_manifest": 0,
> "num_omap_bytes": 0,
> "num_omap_keys": 0,
> "num_objects_repaired": 3
> },
> "up": [
> 208,
> 220,
> 269,
> 175,
> 343,
> 329
> ],
> "acting": [
> 208,
> 220,
> 269,
> 175,
> 313,
> 329
> ],
> "avail_no_missing": [
> "329(5)"
> ],
> "object_location_counts": [
> {
> "shards":
> "175(3),208(0),220(1),269(2),313(4),329(5)",
> "objects": 137710
> },
> {
> "shards": "329(5)",
> "objects": 1
> }
> ],
> "blocked_by": [],
> "up_primary": 208,
> "acting_primary": 208,
> "purged_snaps": []
> },
> "empty": 0,
> "dne": 0,
> "incomplete": 0,
> "last_epoch_started": 141660,
> "hit_set_history": {
> "current_last_update": "0'0",
> "history": []
> }
> },
> {
> "peer": "220(1)",
> "pgid": "404.1ffs1",
> "last_update": "141747'724161",
> "last_complete": "141668'723872",
> "log_tail": "138378'718235",
> "last_user_version": 723769,
> "last_backfill": "MAX",
> "purged_snaps": [],
> "history": {
> "epoch_created": 36041,
> "epoch_pool_created": 27862,
> "last_epoch_started": 141660,
> "last_interval_started": 141659,
> "last_epoch_clean": 140452,
> "last_interval_clean": 140451,
> "last_epoch_split": 108548,
> "last_epoch_marked_full": 0,
> "same_up_since": 141659,
> "same_interval_since": 141659,
> "same_primary_since": 141615,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "prior_readable_until_ub": 0
> },
> "stats": {
> "version": "141624'723769",
> "reported_seq": 3550912,
> "reported_epoch": 141624,
> "state":
"active+recovery_unfound+degraded+remapped",
> "last_fresh": "2023-01-03T11:57:15.480994+0000",
> "last_change":
"2023-01-03T11:31:07.335605+0000",
> "last_active":
"2023-01-03T11:57:15.480994+0000",
> "last_peered":
"2023-01-03T11:57:15.480994+0000",
> "last_clean": "2022-12-31T19:25:42.758665+0000",
> "last_became_active":
> "2023-01-03T11:31:07.116154+0000",
> "last_became_peered":
> "2023-01-03T11:31:07.116154+0000",
> "last_unstale":
"2023-01-03T11:57:15.480994+0000",
> "last_undegraded":
"2023-01-03T11:31:07.099558+0000",
> "last_fullsized":
"2023-01-03T11:57:15.480994+0000",
> "mapping_epoch": 141659,
> "log_start": "138378'718235",
> "ondisk_log_start": "138378'718235",
> "created": 36041,
> "last_epoch_clean": 140452,
> "parent": "0.0",
> "parent_split_bits": 10,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "log_size": 5534,
> "ondisk_log_size": 5534,
> "stats_invalid": false,
> "dirty_stats_invalid": false,
> "omap_stats_invalid": false,
> "hitset_stats_invalid": false,
> "hitset_bytes_stats_invalid": false,
> "pin_stats_invalid": false,
> "manifest_stats_invalid": false,
> "snaptrimq_len": 0,
> "stat_sum": {
> "num_bytes": 281850993538,
> "num_objects": 137711,
> "num_object_clones": 0,
> "num_object_copies": 826266,
> "num_objects_missing_on_primary": 1,
> "num_objects_missing": 1,
> "num_objects_degraded": 5,
> "num_objects_misplaced": 137707,
> "num_objects_unfound": 1,
> "num_objects_dirty": 137711,
> "num_whiteouts": 0,
> "num_read": 720741,
> "num_read_kb": 180044133,
> "num_write": 469883,
> "num_write_kb": 101652239,
> "num_scrub_errors": 0,
> "num_shallow_scrub_errors": 0,
> "num_deep_scrub_errors": 0,
> "num_objects_recovered": 406583,
> "num_bytes_recovered": 891273764570,
> "num_keys_recovered": 0,
> "num_objects_omap": 0,
> "num_objects_hit_set_archive": 0,
> "num_bytes_hit_set_archive": 0,
> "num_flush": 0,
> "num_flush_kb": 0,
> "num_evict": 0,
> "num_evict_kb": 0,
> "num_promote": 0,
> "num_flush_mode_high": 0,
> "num_flush_mode_low": 0,
> "num_evict_mode_some": 0,
> "num_evict_mode_full": 0,
> "num_objects_pinned": 0,
> "num_legacy_snapsets": 0,
> "num_large_omap_objects": 0,
> "num_objects_manifest": 0,
> "num_omap_bytes": 0,
> "num_omap_keys": 0,
> "num_objects_repaired": 3
> },
> "up": [
> 208,
> 220,
> 269,
> 175,
> 343,
> 329
> ],
> "acting": [
> 208,
> 220,
> 269,
> 175,
> 313,
> 329
> ],
> "avail_no_missing": [
> "329(5)"
> ],
> "object_location_counts": [
> {
> "shards":
> "175(3),208(0),220(1),269(2),313(4),329(5)",
> "objects": 137710
> },
> {
> "shards": "329(5)",
> "objects": 1
> }
> ],
> "blocked_by": [],
> "up_primary": 208,
> "acting_primary": 208,
> "purged_snaps": []
> },
> "empty": 0,
> "dne": 0,
> "incomplete": 0,
> "last_epoch_started": 141660,
> "hit_set_history": {
> "current_last_update": "0'0",
> "history": []
> }
> },
> {
> "peer": "269(2)",
> "pgid": "404.1ffs2",
> "last_update": "141747'724161",
> "last_complete": "141668'723872",
> "log_tail": "138378'718235",
> "last_user_version": 723769,
> "last_backfill": "MAX",
> "purged_snaps": [],
> "history": {
> "epoch_created": 36041,
> "epoch_pool_created": 27862,
> "last_epoch_started": 141660,
> "last_interval_started": 141659,
> "last_epoch_clean": 140452,
> "last_interval_clean": 140451,
> "last_epoch_split": 108548,
> "last_epoch_marked_full": 0,
> "same_up_since": 141659,
> "same_interval_since": 141659,
> "same_primary_since": 141615,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "prior_readable_until_ub": 0
> },
> "stats": {
> "version": "141624'723769",
> "reported_seq": 3550912,
> "reported_epoch": 141624,
> "state":
"active+recovery_unfound+degraded+remapped",
> "last_fresh": "2023-01-03T11:57:15.480994+0000",
> "last_change":
"2023-01-03T11:31:07.335605+0000",
> "last_active":
"2023-01-03T11:57:15.480994+0000",
> "last_peered":
"2023-01-03T11:57:15.480994+0000",
> "last_clean": "2022-12-31T19:25:42.758665+0000",
> "last_became_active":
> "2023-01-03T11:31:07.116154+0000",
> "last_became_peered":
> "2023-01-03T11:31:07.116154+0000",
> "last_unstale":
"2023-01-03T11:57:15.480994+0000",
> "last_undegraded":
"2023-01-03T11:31:07.099558+0000",
> "last_fullsized":
"2023-01-03T11:57:15.480994+0000",
> "mapping_epoch": 141659,
> "log_start": "138378'718235",
> "ondisk_log_start": "138378'718235",
> "created": 36041,
> "last_epoch_clean": 140452,
> "parent": "0.0",
> "parent_split_bits": 10,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "log_size": 5534,
> "ondisk_log_size": 5534,
> "stats_invalid": false,
> "dirty_stats_invalid": false,
> "omap_stats_invalid": false,
> "hitset_stats_invalid": false,
> "hitset_bytes_stats_invalid": false,
> "pin_stats_invalid": false,
> "manifest_stats_invalid": false,
> "snaptrimq_len": 0,
> "stat_sum": {
> "num_bytes": 281850993538,
> "num_objects": 137711,
> "num_object_clones": 0,
> "num_object_copies": 826266,
> "num_objects_missing_on_primary": 1,
> "num_objects_missing": 1,
> "num_objects_degraded": 5,
> "num_objects_misplaced": 137707,
> "num_objects_unfound": 1,
> "num_objects_dirty": 137711,
> "num_whiteouts": 0,
> "num_read": 720741,
> "num_read_kb": 180044133,
> "num_write": 469883,
> "num_write_kb": 101652239,
> "num_scrub_errors": 0,
> "num_shallow_scrub_errors": 0,
> "num_deep_scrub_errors": 0,
> "num_objects_recovered": 406583,
> "num_bytes_recovered": 891273764570,
> "num_keys_recovered": 0,
> "num_objects_omap": 0,
> "num_objects_hit_set_archive": 0,
> "num_bytes_hit_set_archive": 0,
> "num_flush": 0,
> "num_flush_kb": 0,
> "num_evict": 0,
> "num_evict_kb": 0,
> "num_promote": 0,
> "num_flush_mode_high": 0,
> "num_flush_mode_low": 0,
> "num_evict_mode_some": 0,
> "num_evict_mode_full": 0,
> "num_objects_pinned": 0,
> "num_legacy_snapsets": 0,
> "num_large_omap_objects": 0,
> "num_objects_manifest": 0,
> "num_omap_bytes": 0,
> "num_omap_keys": 0,
> "num_objects_repaired": 3
> },
> "up": [
> 208,
> 220,
> 269,
> 175,
> 343,
> 329
> ],
> "acting": [
> 208,
> 220,
> 269,
> 175,
> 313,
> 329
> ],
> "avail_no_missing": [
> "329(5)"
> ],
> "object_location_counts": [
> {
> "shards":
> "175(3),208(0),220(1),269(2),313(4),329(5)",
> "objects": 137710
> },
> {
> "shards": "329(5)",
> "objects": 1
> }
> ],
> "blocked_by": [],
> "up_primary": 208,
> "acting_primary": 208,
> "purged_snaps": []
> },
> "empty": 0,
> "dne": 0,
> "incomplete": 0,
> "last_epoch_started": 141660,
> "hit_set_history": {
> "current_last_update": "0'0",
> "history": []
> }
> },
> {
> "peer": "313(4)",
> "pgid": "404.1ffs4",
> "last_update": "141747'724161",
> "last_complete": "141668'723872",
> "log_tail": "138228'717435",
> "last_user_version": 723769,
> "last_backfill": "MAX",
> "purged_snaps": [],
> "history": {
> "epoch_created": 36041,
> "epoch_pool_created": 27862,
> "last_epoch_started": 141660,
> "last_interval_started": 141659,
> "last_epoch_clean": 140452,
> "last_interval_clean": 140451,
> "last_epoch_split": 108548,
> "last_epoch_marked_full": 0,
> "same_up_since": 141659,
> "same_interval_since": 141659,
> "same_primary_since": 141615,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "prior_readable_until_ub": 0
> },
> "stats": {
> "version": "141624'723769",
> "reported_seq": 3551314,
> "reported_epoch": 141659,
> "state":
"active+recovery_unfound+degraded+remapped",
> "last_fresh": "2023-01-03T11:57:15.480994+0000",
> "last_change":
"2023-01-03T11:31:07.335605+0000",
> "last_active":
"2023-01-03T11:57:15.480994+0000",
> "last_peered":
"2023-01-03T11:57:15.480994+0000",
> "last_clean": "2022-12-31T19:25:42.758665+0000",
> "last_became_active":
> "2023-01-03T11:31:07.116154+0000",
> "last_became_peered":
> "2023-01-03T11:31:07.116154+0000",
> "last_unstale":
"2023-01-03T11:57:15.480994+0000",
> "last_undegraded":
"2023-01-03T11:31:07.099558+0000",
> "last_fullsized":
"2023-01-03T11:57:15.480994+0000",
> "mapping_epoch": 141659,
> "log_start": "138378'718235",
> "ondisk_log_start": "138378'718235",
> "created": 36041,
> "last_epoch_clean": 140452,
> "parent": "0.0",
> "parent_split_bits": 10,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "log_size": 5534,
> "ondisk_log_size": 5534,
> "stats_invalid": false,
> "dirty_stats_invalid": false,
> "omap_stats_invalid": false,
> "hitset_stats_invalid": false,
> "hitset_bytes_stats_invalid": false,
> "pin_stats_invalid": false,
> "manifest_stats_invalid": false,
> "snaptrimq_len": 0,
> "stat_sum": {
> "num_bytes": 281850993538,
> "num_objects": 137711,
> "num_object_clones": 0,
> "num_object_copies": 826266,
> "num_objects_missing_on_primary": 1,
> "num_objects_missing": 1,
> "num_objects_degraded": 5,
> "num_objects_misplaced": 137707,
> "num_objects_unfound": 1,
> "num_objects_dirty": 137711,
> "num_whiteouts": 0,
> "num_read": 720741,
> "num_read_kb": 180044133,
> "num_write": 469883,
> "num_write_kb": 101652239,
> "num_scrub_errors": 0,
> "num_shallow_scrub_errors": 0,
> "num_deep_scrub_errors": 0,
> "num_objects_recovered": 406583,
> "num_bytes_recovered": 891273764570,
> "num_keys_recovered": 0,
> "num_objects_omap": 0,
> "num_objects_hit_set_archive": 0,
> "num_bytes_hit_set_archive": 0,
> "num_flush": 0,
> "num_flush_kb": 0,
> "num_evict": 0,
> "num_evict_kb": 0,
> "num_promote": 0,
> "num_flush_mode_high": 0,
> "num_flush_mode_low": 0,
> "num_evict_mode_some": 0,
> "num_evict_mode_full": 0,
> "num_objects_pinned": 0,
> "num_legacy_snapsets": 0,
> "num_large_omap_objects": 0,
> "num_objects_manifest": 0,
> "num_omap_bytes": 0,
> "num_omap_keys": 0,
> "num_objects_repaired": 3
> },
> "up": [
> 208,
> 220,
> 269,
> 175,
> 343,
> 329
> ],
> "acting": [
> 208,
> 220,
> 269,
> 175,
> 313,
> 329
> ],
> "avail_no_missing": [
> "329(5)"
> ],
> "object_location_counts": [
> {
> "shards":
> "175(3),208(0),220(1),269(2),313(4),329(5)",
> "objects": 137710
> },
> {
> "shards": "329(5)",
> "objects": 1
> }
> ],
> "blocked_by": [],
> "up_primary": 208,
> "acting_primary": 208,
> "purged_snaps": []
> },
> "empty": 0,
> "dne": 0,
> "incomplete": 0,
> "last_epoch_started": 141660,
> "hit_set_history": {
> "current_last_update": "0'0",
> "history": []
> }
> },
> {
> "peer": "329(5)",
> "pgid": "404.1ffs5",
> "last_update": "141747'724161",
> "last_complete": "141668'723872",
> "log_tail": "138378'718235",
> "last_user_version": 723769,
> "last_backfill": "MAX",
> "purged_snaps": [],
> "history": {
> "epoch_created": 36041,
> "epoch_pool_created": 27862,
> "last_epoch_started": 141660,
> "last_interval_started": 141659,
> "last_epoch_clean": 140452,
> "last_interval_clean": 140451,
> "last_epoch_split": 108548,
> "last_epoch_marked_full": 0,
> "same_up_since": 141659,
> "same_interval_since": 141659,
> "same_primary_since": 141615,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "prior_readable_until_ub": 0
> },
> "stats": {
> "version": "141624'723769",
> "reported_seq": 3550912,
> "reported_epoch": 141624,
> "state":
"active+recovery_unfound+degraded+remapped",
> "last_fresh": "2023-01-03T11:57:15.480994+0000",
> "last_change":
"2023-01-03T11:31:07.335605+0000",
> "last_active":
"2023-01-03T11:57:15.480994+0000",
> "last_peered":
"2023-01-03T11:57:15.480994+0000",
> "last_clean": "2022-12-31T19:25:42.758665+0000",
> "last_became_active":
> "2023-01-03T11:31:07.116154+0000",
> "last_became_peered":
> "2023-01-03T11:31:07.116154+0000",
> "last_unstale":
"2023-01-03T11:57:15.480994+0000",
> "last_undegraded":
"2023-01-03T11:31:07.099558+0000",
> "last_fullsized":
"2023-01-03T11:57:15.480994+0000",
> "mapping_epoch": 141659,
> "log_start": "138378'718235",
> "ondisk_log_start": "138378'718235",
> "created": 36041,
> "last_epoch_clean": 140452,
> "parent": "0.0",
> "parent_split_bits": 10,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "log_size": 5534,
> "ondisk_log_size": 5534,
> "stats_invalid": false,
> "dirty_stats_invalid": false,
> "omap_stats_invalid": false,
> "hitset_stats_invalid": false,
> "hitset_bytes_stats_invalid": false,
> "pin_stats_invalid": false,
> "manifest_stats_invalid": false,
> "snaptrimq_len": 0,
> "stat_sum": {
> "num_bytes": 281850993538,
> "num_objects": 137711,
> "num_object_clones": 0,
> "num_object_copies": 826266,
> "num_objects_missing_on_primary": 1,
> "num_objects_missing": 0,
> "num_objects_degraded": 5,
> "num_objects_misplaced": 137707,
> "num_objects_unfound": 1,
> "num_objects_dirty": 137711,
> "num_whiteouts": 0,
> "num_read": 720741,
> "num_read_kb": 180044133,
> "num_write": 469883,
> "num_write_kb": 101652239,
> "num_scrub_errors": 0,
> "num_shallow_scrub_errors": 0,
> "num_deep_scrub_errors": 0,
> "num_objects_recovered": 406583,
> "num_bytes_recovered": 891273764570,
> "num_keys_recovered": 0,
> "num_objects_omap": 0,
> "num_objects_hit_set_archive": 0,
> "num_bytes_hit_set_archive": 0,
> "num_flush": 0,
> "num_flush_kb": 0,
> "num_evict": 0,
> "num_evict_kb": 0,
> "num_promote": 0,
> "num_flush_mode_high": 0,
> "num_flush_mode_low": 0,
> "num_evict_mode_some": 0,
> "num_evict_mode_full": 0,
> "num_objects_pinned": 0,
> "num_legacy_snapsets": 0,
> "num_large_omap_objects": 0,
> "num_objects_manifest": 0,
> "num_omap_bytes": 0,
> "num_omap_keys": 0,
> "num_objects_repaired": 3
> },
> "up": [
> 208,
> 220,
> 269,
> 175,
> 343,
> 329
> ],
> "acting": [
> 208,
> 220,
> 269,
> 175,
> 313,
> 329
> ],
> "avail_no_missing": [
> "329(5)"
> ],
> "object_location_counts": [
> {
> "shards":
> "175(3),208(0),220(1),269(2),313(4),329(5)",
> "objects": 137710
> },
> {
> "shards": "329(5)",
> "objects": 1
> }
> ],
> "blocked_by": [],
> "up_primary": 208,
> "acting_primary": 208,
> "purged_snaps": []
> },
> "empty": 0,
> "dne": 0,
> "incomplete": 0,
> "last_epoch_started": 141660,
> "hit_set_history": {
> "current_last_update": "0'0",
> "history": []
> }
> },
> {
> "peer": "343(4)",
> "pgid": "404.1ffs4",
> "last_update": "141747'724161",
> "last_complete": "141668'723872",
> "log_tail": "138378'718235",
> "last_user_version": 723769,
> "last_backfill":
>
"404:ff80001b:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-09T12%3a41%3a05,478440857+00%3a00.2~XwOh1_IhVzghlM-zi7DfwoUIZl5UxVt.1:head",
> "purged_snaps": [],
> "history": {
> "epoch_created": 36041,
> "epoch_pool_created": 27862,
> "last_epoch_started": 141660,
> "last_interval_started": 141659,
> "last_epoch_clean": 140452,
> "last_interval_clean": 140451,
> "last_epoch_split": 108548,
> "last_epoch_marked_full": 0,
> "same_up_since": 141659,
> "same_interval_since": 141659,
> "same_primary_since": 141615,
> "last_scrub": "140525'722168",
> "last_scrub_stamp":
"2022-12-31T19:27:10.993286+0000",
> "last_deep_scrub": "140525'722168",
> "last_deep_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "last_clean_scrub_stamp":
> "2022-12-31T19:27:10.993286+0000",
> "prior_readable_until_ub": 0
> },
> "stats": {
> "version": "0'0",
> "reported_seq": 0,
> "reported_epoch": 0,
> "state": "unknown",
> "last_fresh": "0.000000",
> "last_change": "0.000000",
> "last_active": "0.000000",
> "last_peered": "0.000000",
> "last_clean": "0.000000",
> "last_became_active": "0.000000",
> "last_became_peered": "0.000000",
> "last_unstale": "0.000000",
> "last_undegraded": "0.000000",
> "last_fullsized": "0.000000",
> "mapping_epoch": 141659,
> "log_start": "0'0",
> "ondisk_log_start": "0'0",
> "created": 0,
> "last_epoch_clean": 0,
> "parent": "0.0",
> "parent_split_bits": 0,
> "last_scrub": "0'0",
> "last_scrub_stamp": "0.000000",
> "last_deep_scrub": "0'0",
> "last_deep_scrub_stamp": "0.000000",
> "last_clean_scrub_stamp": "0.000000",
> "log_size": 0,
> "ondisk_log_size": 0,
> "stats_invalid": false,
> "dirty_stats_invalid": false,
> "omap_stats_invalid": false,
> "hitset_stats_invalid": false,
> "hitset_bytes_stats_invalid": false,
> "pin_stats_invalid": false,
> "manifest_stats_invalid": false,
> "snaptrimq_len": 0,
> "stat_sum": {
> "num_bytes": 9404416,
> "num_objects": 3,
> "num_object_clones": 0,
> "num_object_copies": 0,
> "num_objects_missing_on_primary": 0,
> "num_objects_missing": 137907,
> "num_objects_degraded": 0,
> "num_objects_misplaced": 0,
> "num_objects_unfound": 0,
> "num_objects_dirty": 3,
> "num_whiteouts": 0,
> "num_read": 0,
> "num_read_kb": 0,
> "num_write": 0,
> "num_write_kb": 0,
> "num_scrub_errors": 0,
> "num_shallow_scrub_errors": 0,
> "num_deep_scrub_errors": 0,
> "num_objects_recovered": 0,
> "num_bytes_recovered": 0,
> "num_keys_recovered": 0,
> "num_objects_omap": 0,
> "num_objects_hit_set_archive": 0,
> "num_bytes_hit_set_archive": 0,
> "num_flush": 0,
> "num_flush_kb": 0,
> "num_evict": 0,
> "num_evict_kb": 0,
> "num_promote": 0,
> "num_flush_mode_high": 0,
> "num_flush_mode_low": 0,
> "num_evict_mode_some": 0,
> "num_evict_mode_full": 0,
> "num_objects_pinned": 0,
> "num_legacy_snapsets": 0,
> "num_large_omap_objects": 0,
> "num_objects_manifest": 0,
> "num_omap_bytes": 0,
> "num_omap_keys": 0,
> "num_objects_repaired": 0
> },
> "up": [
> 208,
> 220,
> 269,
> 175,
> 343,
> 329
> ],
> "acting": [
> 208,
> 220,
> 269,
> 175,
> 313,
> 329
> ],
> "avail_no_missing": [],
> "object_location_counts": [],
> "blocked_by": [],
> "up_primary": 208,
> "acting_primary": 208,
> "purged_snaps": []
> },
> "empty": 0,
> "dne": 0,
> "incomplete": 1,
> "last_epoch_started": 141660,
> "hit_set_history": {
> "current_last_update": "0'0",
> "history": []
> }
> }
> ],
> "recovery_state": [
> {
> "name": "Started/Primary/Active",
> "enter_time": "2023-01-03T12:15:49.758685+0000",
> "might_have_unfound": [
> {
> "osd": "175(3)",
> "status": "already probed"
> },
> {
> "osd": "220(1)",
> "status": "already probed"
> },
> {
> "osd": "269(2)",
> "status": "already probed"
> },
> {
> "osd": "313(4)",
> "status": "already probed"
> },
> {
> "osd": "329(5)",
> "status": "already probed"
> },
> {
> "osd": "343(4)",
> "status": "already probed"
> }
> ],
> "recovery_progress": {
> "backfill_targets": [
> "343(4)"
> ],
> "waiting_on_backfill": [],
> "last_backfill_started":
>
"404:ff80001b:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-09T12%3a41%3a05,478440857+00%3a00.2~XwOh1_IhVzghlM-zi7DfwoUIZl5UxVt.1:head",
> "backfill_info": {
> "begin": "MIN",
> "end": "MIN",
> "objects": []
> },
> "peer_backfill_info": [],
> "backfills_in_flight": [],
> "recovering": [],
> "pg_backend": {
> "recovery_ops": [],
> "read_ops": []
> }
> }
> },
> {
> "name": "Started",
> "enter_time": "2023-01-03T12:15:48.766366+0000"
> }
> ],
> "scrubber": {
> "epoch_start": "0",
> "active": false
> },
> "agent_state": {}
> }
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io