Working ceph cluster reports large amount of pgs in state unknown/undersized and objects degraded
We operate a tiny ceph cluster (v16.2.7) across three machines, each running two OSDs and one of each mds, mgr, and mon. The cluster serves one main erasure-coded (2+1) storage pool and a few other management-related pools. The cluster has been running smoothly for several months. A few weeks ago we noticed a health warning reporting backfillfull/nearfull osds and pools. Here is the output of `ceph -s` at this point (extraced from logs): -------------------------------------------------------------------------------- cluster: health: HEALTH_WARN 1 backfillfull osd(s) 2 nearfull osd(s) Reduced data availability: 163 pgs inactive, 1 pg peering Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull Degraded data redundancy: 1486709/10911157 objects degraded (13.626%), 68 pgs degraded, 68 pgs undersized 162 pgs not scrubbed in time 6 pool(s) backfillfull services: mon: 3 daemons, quorum mon.101,mon.102,mon.100 (age 5m) mgr: mgr-102(active, since 54m), standbys: mgr-101, mgr-100 mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 6 osds: 6 up (since 4m), 6 in (since 2w); 7 remapped pgs data: volumes: 1/1 healthy pools: 6 pools, 338 pgs objects: 3.64M objects, 14 TiB usage: 13 TiB used, 1.7 TiB / 15 TiB avail pgs: 47.929% pgs unknown 0.296% pgs not active 1486709/10911157 objects degraded (13.626%) 52771/10911157 objects misplaced (0.484%) 162 unknown 106 active+clean 67 active+undersized+degraded 1 active+undersized+degraded+remapped+backfill_toofull 1 remapped+peering 1 active+remapped+backfill_toofull -------------------------------------------------------------------------------- I now see the large amount of pgs in state unknown and the fact that a significant fraction of objects is degraded despite all osds being up, but we didn't notice this back then. Because the cluster continued to act fine from the perspective of the mounted filesystem, we didn't really notice the potential problem and did not intervene. From then one, things have mostly gone downwards. Now, `ceph -s` reports the following: -------------------------------------------------------------------------------- cluster: health: HEALTH_WARN noout flag(s) set Reduced data availability: 117 pgs inactive Degraded data redundancy: 2095625/12121767 objects degraded (17.288%), 114 pgs degraded, 114 pgs undersized 117 pgs not scrubbed in time services: mon: 3 daemons, quorum mon.101,mon.102,mon.100 (age 15h) mgr: mgr-102(active, since 7d), standbys: mgr-100, mgr-101 mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 6 osds: 6 up (since 55m), 6 in (since 3w) flags noout data: volumes: 1/1 healthy pools: 6 pools, 338 pgs objects: 4.04M objects, 15 TiB usage: 12 TiB used, 2.8 TiB / 15 TiB avail pgs: 34.615% pgs unknown 2095625/12121767 objects degraded (17.288%) 117 unknown 114 active+undersized+degraded 107 active+clean -------------------------------------------------------------------------------- Note in particular the still very large number of pgs in state unknown, which hasn't changed in days. Same goes for the degraded pgs. Also, the cluster should have around 37TiB storage available but now it only reports 15 TiB. We did a bit of digging around but couldn't really get to the bottom of the unknown pgs and how we can recover from that. One other data point is that the command `ceph osd df tree` gets stuck on two of the three machines and one the one where it returns something, it looks like this: -------------------------------------------------------------------------------- ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 47.67506 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root default -13 18.26408 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - datacenter dc.100 -5 18.26408 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host osd-100 3 hdd 10.91409 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 91 up osd.3 5 hdd 7.34999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 48 up osd.5 -9 14.69998 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - datacenter dc.101 -7 14.69998 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host osd-101 0 hdd 7.34999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 83 up osd.0 1 hdd 7.34999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 86 up osd.1 -11 14.71100 - 15 TiB 12 TiB 12 TiB 77 MiB 21 GiB 2.6 TiB 82.00 1.00 - datacenter dc.102 -17 7.35550 - 7.4 TiB 6.3 TiB 6.2 TiB 16 MiB 11 GiB 1.1 TiB 85.16 1.04 - host osdroid-102-1 4 hdd 7.35550 1.00000 7.4 TiB 6.3 TiB 6.2 TiB 16 MiB 11 GiB 1.1 TiB 85.16 1.04 114 up osd.4 -15 7.35550 - 7.4 TiB 5.8 TiB 5.7 TiB 61 MiB 10 GiB 1.6 TiB 78.83 0.96 - host osdroid-102-2 2 hdd 7.35550 1.00000 7.4 TiB 5.8 TiB 5.7 TiB 61 MiB 10 GiB 1.6 TiB 78.83 0.96 107 up osd.2 TOTAL 15 TiB 12 TiB 12 TiB 77 MiB 21 GiB 2.6 TiB 82.00 MIN/MAX VAR: 0/1.04 STDDEV: 66.97 -------------------------------------------------------------------------------- The odd part here is that for some reason only osd.2 and osd.4 seem to contribute size to the cluster. Interestingly, accessing content from the storage pool works mostly without issues, which shouldn't work if 4 out of 6 OSDs weren't properly up. Even more odd is that while `ceph health detail` reports a lot of pgs in state unknown, undersized, and degraded, inspecting the respective pgs with `ceph pg <pdid> query` results in active+clean for *all* of them... I'm not sure which of the two pieces of information I am supposed to trust... Any ideas what we can do to get our cluster back into a sane state? I'm happy to provide more logs or command output, please let me know. Thanks!
Hi Tobias, April 18, 2024 at 8:08 PM, "Tobias Langner" <tlangner+ceph@bitvalve.org> wrote:
We operate a tiny ceph cluster (v16.2.7) across three machines, each
running two OSDs and one of each mds, mgr, and mon. The cluster serves
one main erasure-coded (2+1) storage pool and a few other
I'd assume (w/o pool config) that the EC 2+1 is putting PG as inactive. Because for EC you need n-2 for redundancy and n-1 for availability. The output got a bit mangled. Could you please provide them in some pastebin maybe? Can you please post the crush rule and pool settings? To better understand the data distribution. And what does the logs show on one of the affected OSDs? Cheers, Alwin
Hey Alwin, Thanks for your reply, answers inline.
I'd assume (w/o pool config) that the EC 2+1 is putting PG as inactive. Because for EC you need n-2 for redundancy and n-1 for availability. Yeah, I guess this is likely related to the issue.
The output got a bit mangled. Could you please provide them in some pastebin maybe? Yes of course, I hoped that my mail would be displayed with monospace font by default :)
Here are the outputs: `ceph -s` when we first noticed something was wrong: https://pastebin.com/raw/FcUFB25D `ceph -s` now: https://pastebin.com/raw/rsLynw2V `ceph osd df tree`: https://pastebin.com/raw/kD9VEcLR
Can you please post the crush rule and pool settings? To better understand the data distribution. And what does the logs show on one of the affected OSDs? `ceph osd crush dump`: https://pastebin.com/raw/DWEHcNaA
While trying to dig up a bit more information, I noticed that the mgr web UI was down, which is why we failed the active mgr to have one of the standbys to take over, without thinking much... Lo and behold, this completely resolved the issue from one moment to the other. Now `ceph -s` return 338 active+clean pgs, as expected and desired... While we are naturally pretty happy that the problem resolved itself, it would still be good to understand 1. what caused this weird state in which `ceph -s` output did not match what was happening in reality, 2. how a mgr failover could cause changes in `ceph -s` output, thereby fixing the above issue, 3. why `ceph osd df tree` reported a weird split state with only few hosts contributing storage, and why it did not return any results when run on some other hosts. Happy to hear your thoughts!
Hi Tobias, April 18, 2024 at 10:43 PM, "Tobias Langner" <tlangner+ceph@bitvalve.org> wrote:
While trying to dig up a bit more information, I noticed that the mgr web UI was down, which is why we failed the active mgr to have one of the standbys to take over, without thinking much...
Lo and behold, this completely resolved the issue from one moment to the other. Now `ceph -s` return 338 active+clean pgs, as expected and desired...
While we are naturally pretty happy that the problem resolved itself, it would still be good to understand Thank you that confirms my thought.
1. what caused this weird state in which `ceph -s` output did not match
The MGR provides the stats for it.
2. how a mgr failover could cause changes in `ceph -s` output, thereby
See above.
3. why `ceph osd df tree` reported a weird split state with only few
Likely the same. You'd need to go through the MGR log and see what caused the MGR to hang. Cheers, Alwin
Hi, there are lots of metrics that are collected by the MGR. So if there is none, the cluster health details can be wrong or outdated. Zitat von Tobias Langner <tlangner+ceph@bitvalve.org>:
Hey Alwin,
Thanks for your reply, answers inline.
I'd assume (w/o pool config) that the EC 2+1 is putting PG as inactive. Because for EC you need n-2 for redundancy and n-1 for availability. Yeah, I guess this is likely related to the issue.
The output got a bit mangled. Could you please provide them in some pastebin maybe? Yes of course, I hoped that my mail would be displayed with monospace font by default :)
Here are the outputs:
`ceph -s` when we first noticed something was wrong: https://pastebin.com/raw/FcUFB25D
`ceph -s` now: https://pastebin.com/raw/rsLynw2V
`ceph osd df tree`: https://pastebin.com/raw/kD9VEcLR
Can you please post the crush rule and pool settings? To better understand the data distribution. And what does the logs show on one of the affected OSDs? `ceph osd crush dump`: https://pastebin.com/raw/DWEHcNaA
While trying to dig up a bit more information, I noticed that the mgr web UI was down, which is why we failed the active mgr to have one of the standbys to take over, without thinking much...
Lo and behold, this completely resolved the issue from one moment to the other. Now `ceph -s` return 338 active+clean pgs, as expected and desired...
While we are naturally pretty happy that the problem resolved itself, it would still be good to understand
1. what caused this weird state in which `ceph -s` output did not match what was happening in reality, 2. how a mgr failover could cause changes in `ceph -s` output, thereby fixing the above issue, 3. why `ceph osd df tree` reported a weird split state with only few hosts contributing storage, and why it did not return any results when run on some other hosts.
Happy to hear your thoughts! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Some additional information. Even though the pgs report unknown, directly querying them seems like they are correctly up. What could be causing this disconnect in reported pg states? ``` $ ceph pg dump_stuck inactive | head ok PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 16.fc unknown [] -1 [] -1 16.fb unknown [] -1 [] -1 16.fa unknown [] -1 [] -1 ... ... ... ``` ``` $ ceph osd dump | grep epoch epoch 54661 ``` ``` $ ceph pg 16.fc query { "snap_trimq": "[]", "snap_trimq_len": 0, "state": "active+clean", "epoch": 54661, "up": [ 5, 0, 4 ], "acting": [ 5, 0, 4 ], "acting_recovery_backfill": [ "0(1)", "4(2)", "5(0)" ], "info": { "pgid": "16.fcs0", "last_update": "54599'266234", "last_complete": "54599'266234", "log_tail": "54406'263772", "last_user_version": 266234, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 14243, "epoch_pool_created": 2798, "last_epoch_started": 54660, "last_interval_started": 54659, "last_epoch_clean": 54660, "last_interval_clean": 54659, "last_epoch_split": 14243, "last_epoch_marked_full": 24149, "same_up_since": 54659, "same_interval_since": 54659, "same_primary_since": 54586, "last_scrub": "54579'266179", "last_scrub_stamp": "2024-04-18T01:20:28.226815+0200", "last_deep_scrub": "53783'261235", "last_deep_scrub_stamp": "2024-03-26T03:29:22.874529+0100", "last_clean_scrub_stamp": "2024-04-18T01:20:28.226815+0200", "prior_readable_until_ub": 0 }, "stats": { "version": "54599'266234", "reported_seq": 1432180, "reported_epoch": 54661, "state": "active+clean", "last_fresh": "2024-04-18T21:09:57.484866+0200", "last_change": "2024-04-18T20:57:24.855741+0200", "last_active": "2024-04-18T21:09:57.484866+0200", "last_peered": "2024-04-18T21:09:57.484866+0200", "last_clean": "2024-04-18T21:09:57.484866+0200", "last_became_active": "2024-04-18T20:57:24.855556+0200", "last_became_peered": "2024-04-18T20:57:24.855556+0200", "last_unstale": "2024-04-18T21:09:57.484866+0200", "last_undegraded": "2024-04-18T21:09:57.484866+0200", "last_fullsized": "2024-04-18T21:09:57.484866+0200", "mapping_epoch": 54659, "log_start": "54406'263772", "ondisk_log_start": "54406'263772", "created": 14243, "last_epoch_clean": 54660, "parent": "0.0", "parent_split_bits": 8, "last_scrub": "54579'266179", "last_scrub_stamp": "2024-04-18T01:20:28.226815+0200", "last_deep_scrub": "53783'261235", "last_deep_scrub_stamp": "2024-03-26T03:29:22.874529+0100", "last_clean_scrub_stamp": "2024-04-18T01:20:28.226815+0200", "log_size": 2462, "ondisk_log_size": 2462, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 100198672069, "num_objects": 24008, "num_object_clones": 0, "num_object_copies": 72024, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 24008, "num_whiteouts": 0, "num_read": 481265, "num_read_kb": 589512788, "num_write": 160947, "num_write_kb": 164785104, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 132312, "num_bytes_recovered": 548853441237, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 5, 0, 4 ], "acting": [ 5, 0, 4 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 5, "acting_primary": 5, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 54660, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "0(1)", "pgid": "16.fcs1", "last_update": "54599'266234", "last_complete": "54599'266234", "log_tail": "54406'263772", "last_user_version": 266234, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 14243, "epoch_pool_created": 2798, "last_epoch_started": 54660, "last_interval_started": 54659, "last_epoch_clean": 54660, "last_interval_clean": 54659, "last_epoch_split": 14243, "last_epoch_marked_full": 24149, "same_up_since": 54659, "same_interval_since": 54659, "same_primary_since": 54586, "last_scrub": "54579'266179", "last_scrub_stamp": "2024-04-18T01:20:28.226815+0200", "last_deep_scrub": "53783'261235", "last_deep_scrub_stamp": "2024-03-26T03:29:22.874529+0100", "last_clean_scrub_stamp": "2024-04-18T01:20:28.226815+0200", "prior_readable_until_ub": 0 }, "stats": { "version": "54599'266234", "reported_seq": 1432105, "reported_epoch": 54599, "state": "active+clean", "last_fresh": "2024-04-18T16:59:22.048690+0200", "last_change": "2024-04-18T04:12:58.494926+0200", "last_active": "2024-04-18T16:59:22.048690+0200", "last_peered": "2024-04-18T16:59:22.048690+0200", "last_clean": "2024-04-18T16:59:22.048690+0200", "last_became_active": "2024-04-18T04:12:43.776065+0200", "last_became_peered": "2024-04-18T04:12:43.776065+0200", "last_unstale": "2024-04-18T16:59:22.048690+0200", "last_undegraded": "2024-04-18T16:59:22.048690+0200", "last_fullsized": "2024-04-18T16:59:22.048690+0200", "mapping_epoch": 54659, "log_start": "54406'263772", "ondisk_log_start": "54406'263772", "created": 14243, "last_epoch_clean": 54587, "parent": "0.0", "parent_split_bits": 8, "last_scrub": "54579'266179", "last_scrub_stamp": "2024-04-18T01:20:28.226815+0200", "last_deep_scrub": "53783'261235", "last_deep_scrub_stamp": "2024-03-26T03:29:22.874529+0100", "last_clean_scrub_stamp": "2024-04-18T01:20:28.226815+0200", "log_size": 2462, "ondisk_log_size": 2462, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 100198672069, "num_objects": 24008, "num_object_clones": 0, "num_object_copies": 72024, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 24008, "num_whiteouts": 0, "num_read": 481265, "num_read_kb": 589512788, "num_write": 160947, "num_write_kb": 164785104, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 132312, "num_bytes_recovered": 548853441237, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 5, 0, 4 ], "acting": [ 5, 0, 4 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 5, "acting_primary": 5, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 54660, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "4(2)", "pgid": "16.fcs2", "last_update": "54599'266234", "last_complete": "54599'266234", "log_tail": "54406'263772", "last_user_version": 266234, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 14243, "epoch_pool_created": 2798, "last_epoch_started": 54660, "last_interval_started": 54659, "last_epoch_clean": 54660, "last_interval_clean": 54659, "last_epoch_split": 14243, "last_epoch_marked_full": 24149, "same_up_since": 54659, "same_interval_since": 54659, "same_primary_since": 54586, "last_scrub": "54579'266179", "last_scrub_stamp": "2024-04-18T01:20:28.226815+0200", "last_deep_scrub": "53783'261235", "last_deep_scrub_stamp": "2024-03-26T03:29:22.874529+0100", "last_clean_scrub_stamp": "2024-04-18T01:20:28.226815+0200", "prior_readable_until_ub": 0 }, "stats": { "version": "54599'266234", "reported_seq": 1432105, "reported_epoch": 54599, "state": "active+clean", "last_fresh": "2024-04-18T16:59:22.048690+0200", "last_change": "2024-04-18T04:12:58.494926+0200", "last_active": "2024-04-18T16:59:22.048690+0200", "last_peered": "2024-04-18T16:59:22.048690+0200", "last_clean": "2024-04-18T16:59:22.048690+0200", "last_became_active": "2024-04-18T04:12:43.776065+0200", "last_became_peered": "2024-04-18T04:12:43.776065+0200", "last_unstale": "2024-04-18T16:59:22.048690+0200", "last_undegraded": "2024-04-18T16:59:22.048690+0200", "last_fullsized": "2024-04-18T16:59:22.048690+0200", "mapping_epoch": 54659, "log_start": "54406'263772", "ondisk_log_start": "54406'263772", "created": 14243, "last_epoch_clean": 54587, "parent": "0.0", "parent_split_bits": 8, "last_scrub": "54579'266179", "last_scrub_stamp": "2024-04-18T01:20:28.226815+0200", "last_deep_scrub": "53783'261235", "last_deep_scrub_stamp": "2024-03-26T03:29:22.874529+0100", "last_clean_scrub_stamp": "2024-04-18T01:20:28.226815+0200", "log_size": 2462, "ondisk_log_size": 2462, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 100198672069, "num_objects": 24008, "num_object_clones": 0, "num_object_copies": 72024, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 24008, "num_whiteouts": 0, "num_read": 481265, "num_read_kb": 589512788, "num_write": 160947, "num_write_kb": 164785104, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 132312, "num_bytes_recovered": 548853441237, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 5, 0, 4 ], "acting": [ 5, 0, 4 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 5, "acting_primary": 5, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 54660, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2024-04-18T20:57:24.774643+0200", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "recovery_ops": [], "read_ops": [] } } }, { "name": "Started", "enter_time": "2024-04-18T20:57:23.760477+0200" } ], "scrubber": { "epoch_start": "54566", "active": false }, "agent_state": {} } ```
participants (4)
-
Alwin Antreich
-
c+gvihgmke＠picton.nom.za
-
Eugen Block
-
Tobias Langner