- ceph-users - lists.ceph.io

[ERR] OSD_SCRUB_ERRORS: 2 scrub errors
by Szabo, Istvan (Agoda) 11 Jan '23

11 Jan '23

Hi, I’m continuously getting scrub errors in my index pool and log pool that I need to repair always. HEALTH_ERR 2 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 2 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 20.19 is active+clean+inconsistent, acting [39,41,37] Why is this? I have no cue at all, no log entry no anything ☹ ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

6 9

Intel Cache Solution with HA Cluster on the iSCSI Gateway node
by Kamran Zafar Syed 11 Jan '23

11 Jan '23

HI There, Is there someone, who had some experience of implementing Intel Cache Accelerator Solution on top of iSCSI Gateway. Thanks and Regards, Koki

1 0

nfs RGW export makes nfs-gnaesha server in crash loop
by Ben 11 Jan '23

11 Jan '23

Hi, This is running Quincy 17.2.5 deployed by rook on k8s. RGW nfs export will crash Ganesha server pod. CephFS export works just fine. Here are steps of it: 1, create export: bash-4.4$ ceph nfs export create rgw --cluster-id nfs4rgw --pseudo-path /bucketexport --bucket testbk { "bind": "/bucketexport", "path": "testbk", "cluster": "nfs4rgw", "mode": "RW", "squash": "none" } 2, check pods status afterwards: rook-ceph-nfs-nfs1-a-679fdb795-82tcx 2/2 Running 0 4h3m rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 1/2 Error 2 4h6m 3, check failing pod’s logs: 11/01/2023 08:11:53 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 11/01/2023 08:11:54 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend 11/01/2023 08:11:54 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) 11/01/2023 08:11:57 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE 11/01/2023 08:11:57 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] export_defaults_commit :CONFIG :INFO :Export Defaults now (options=03303002/00080000 , , , , , , , , expire= 0) 2023-01-11T08:11:57.853+0000 7f59dac7c200 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-admin/keyring: (2) No such file or directory 2023-01-11T08:11:57.853+0000 7f59dac7c200 -1 AuthRegistry(0x56476817a480) no keyring found at /var/lib/ceph/radosgw/ceph-admin/keyring, disabling cephx 2023-01-11T08:11:57.855+0000 7f59dac7c200 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-admin/keyring: (2) No such file or directory 2023-01-11T08:11:57.855+0000 7f59dac7c200 -1 AuthRegistry(0x7ffe4d092c90) no keyring found at /var/lib/ceph/radosgw/ceph-admin/keyring, disabling cephx 2023-01-11T08:11:57.856+0000 7f5987537700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2023-01-11T08:11:57.856+0000 7f5986535700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2023-01-11T08:12:00.861+0000 7f5986d36700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2023-01-11T08:12:00.861+0000 7f59dac7c200 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication failed to fetch mon config (--no-mon-config to skip) 4, delete the export: ceph nfs export delete nfs4rgw /bucketexport Ganesha servers go back normal: rook-ceph-nfs-nfs1-a-679fdb795-82tcx 2/2 Running 0 4h30m rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 2/2 Running 10 4h33m Any ideas to make it work? Thanks Ben

1 0

Issues with cephadm adopt cluster with name
by armsby 11 Jan '23

11 Jan '23

I am trying to adopt a cluster with cephadm, and everything was ok when it came to mon and mgr servers, But when I try to run "cephadm adopt --name osd.340 --style legacy —cluster prod" It runs everything, but when the container starts, it says that it can not open /etc/ceph/prod.conf as it binds config in as /etc/ceph/ceph.conf If I change the unit.run file so it mounts it in as prod.conf it starts but has issues connecting to mon servers. Has someone else experienced this, and are there solutions for this issue? Armsby

1 0

2 pgs backfill_toofull but plenty of space
by Torkil Svensgaard 10 Jan '23

10 Jan '23

Hi Ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable) Looking at this: " Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull " " [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull pg 3.11f is active+remapped+backfill_wait+backfill_toofull, acting [98,51,39,100] pg 3.74c is active+remapped+backfill_wait+backfill_toofull, acting [96,120,58,48] " But the disks are noway near being full as far as I can determine, so why backfill_toofull? The PGs in question are in the rbd_data pool. " # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1.4 PiB 730 TiB 686 TiB 686 TiB 48.46 ssd 1.3 TiB 1.2 TiB 162 GiB 162 GiB 12.11 TOTAL 1.4 PiB 731 TiB 686 TiB 686 TiB 48.42 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 1.1 GiB 273 545 MiB 0.05 549 GiB rbd_data 3 4096 294 TiB 78.56M 450 TiB 45.72 267 TiB rbd 4 32 4.1 MiB 26 3.5 MiB 0 549 GiB rbd_internal 5 32 54 KiB 16 172 KiB 0 549 GiB cephfs_data 6 2048 127 TiB 148.64M 229 TiB 29.99 267 TiB cephfs_metadata 7 128 71 GiB 2.84M 142 GiB 11.46 549 GiB libvirt 8 32 37 MiB 221 74 MiB 0 549 GiB nfs-ganesha 9 32 2.7 KiB 7 52 KiB 0 366 GiB .nfs 10 32 53 KiB 47 306 KiB 0 366 GiB " The top utilized disk is at 57% and the PGs in that pool are ~50GB. " TOP BOTTOM USE WEIGHT PGS ID |USE WEIGHT PGS ID --------------------------------+-------------------------------- 57.71% 1.00000 54 osd.68 |46.60% 1.00000 286 osd.17 57.08% 1.00000 53 osd.80 |46.55% 1.00000 286 osd.99 54.95% 1.00000 70 osd.86 |46.48% 1.00000 284 osd.106 54.86% 1.00000 52 osd.63 |45.88% 1.00000 187 osd.27 54.06% 1.00000 68 osd.88 |45.81% 1.00000 279 osd.5 53.89% 1.00000 51 osd.79 |44.95% 1.00000 272 osd.13 53.65% 1.00000 51 osd.67 |43.63% 1.00000 269 osd.16 53.59% 1.00000 52 osd.65 |43.30% 1.00000 261 osd.12 53.58% 1.00000 51 osd.82 |32.17% 1.00000 172 osd.4 53.52% 1.00000 50 osd.72 |0% 0 0 osd.49 --------------------------------+-------------------------------- " Mvh. Torkil -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

3 2

Re: What's happening with ceph-users?
by Neha Ojha 10 Jan '23

10 Jan '23

Re-adding the dev list and adding the user list because others might benefit from this information. Thanks, Neha On Tue, Jan 10, 2023 at 10:21 AM Wyll Ingersoll < wyllys.ingersoll(a)keepertech.com> wrote: > Also, it was only my ceph-users account that was lost, dev account was > still active. > ------------------------------ > *From:* Wyll Ingersoll <wyllys.ingersoll(a)keepertech.com> > *Sent:* Tuesday, January 10, 2023 1:20 PM > *To:* Neha Ojha <nojha(a)redhat.com>; Adam Kraitman <akraitma(a)redhat.com>; > Dan Mick <dan.mick(a)redhat.com> > *Subject:* Re: What's happening with ceph-users? > > I ended up re-subscribing this morning. But it might be worth > investigating if others are having similar issues. > ------------------------------ > *From:* Neha Ojha <nojha(a)redhat.com> > *Sent:* Tuesday, January 10, 2023 1:14 PM > *To:* Wyll Ingersoll <wyllys.ingersoll(a)keepertech.com>; Adam Kraitman < > akraitma(a)redhat.com>; Dan Mick <dan.mick(a)redhat.com> > *Subject:* Re: What's happening with ceph-users? > > +Adam Kraitman <akraitma(a)redhat.com> +Dan Mick <dan.mick(a)redhat.com> Is > this expected? > > On Tue, Jan 10, 2023 at 6:15 AM Wyll Ingersoll < > wyllys.ingersoll(a)keepertech.com> wrote: > > > All of my subscriptions to the ceph.io lists (users and developers) seem > to have been deleted. Do we need to re-subscribe or is this something > that is being fixed? > ------------------------------ > *From:* Neha Ojha <nojha(a)redhat.com> > *Sent:* Monday, January 9, 2023 2:40 PM > *To:* Dan van der Ster <dvanders(a)gmail.com> > *Cc:* Ceph Developers <dev(a)ceph.io>; Josh Durgin <jdurgin(a)redhat.com>; > Mike Perez <miperez(a)redhat.com>; Adam Kraitman <akraitma(a)redhat.com> > *Subject:* Re: What's happening with ceph-users? > > Our mailing lists were down due to the recent lab issues. They should be > back up now. Please let us know if you see any issues. > > Thanks, > Neha > > On Sun, Jan 8, 2023 at 9:53 AM Dan van der Ster <dvanders(a)gmail.com> > wrote: > > Hi, > > Has ceph-users been down a few days? And now it seems to have been > reverted to an old backup? (I'm referring mail on an address I unsubbed > many months ago) > > Thanks, Dan > >

1 0

Re: docs.ceph.com -- Do you use the header navigation bar? (RESPONSES REQUESTED)
by Frank Schilder 10 Jan '23

10 Jan '23

Hi John, firstly, image attachments are filtered out by the list. How about you upload the image somewhere like https://imgur.com/ and post a link instead? In my browser, the sticky header contains only "home" and "edit on github", which are both entirely useless for a user. What exactly is "header navigation" expected to do if it contains nothing else? Unless I'm looking at the wrong thing (I can't see the attached image), this header can be removed. The "edit on github" link can be added to the end of a page. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: John Zachary Dover <zac.dover(a)gmail.com> Sent: 04 January 2023 16:35:56 To: ceph-users Subject: [ceph-users] docs.ceph.com -- Do you use the header navigation bar? (RESPONSES REQUESTED) Do you use the header navigation bar on docs.ceph.com? See the attached file (sticky_header.png) if you are unsure of what "header navigation bar" means. In the attached file, the header navigation bar is indicated by means of two large, ugly, red-and-green arrows. *Cards on the Table* The navigation bar is the kind of thing that is sometimes referred to as a "sticky header", and it can get in the way of linked-to sections. I would like to remove this header bar. If there is community support for the header bar, though, I won't remove it. *What is Zac Complaining About?* Follow this procedure to see the behavior that has provoked my complaint: 1. Go to https://docs.ceph.com/en/quincy/glossary/ 2. Scroll down to the "Ceph Cluster Map" entry. 3. Click the "Cluster Map" link in the line that reads "See Cluster Map". 4. Notice that the header navigation bar obscures the headword "Cluster Map". If you have any opinion at all on this matter, voice it. Please. Zac Dover Docs Upstream Ceph _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Re: rbd-mirror stops replaying journal on primary cluster
by Josef Johansson 10 Jan '23

10 Jan '23

Hi, Actually, the test case was even more simple than that. A misaligned discard (discard_granularity_bytes=4096, offset=0, length=4096+512) made the journal stop replaying entries. This is now well covered in tests and example e2e-tests. The workaround is quite easy, set `rbd_discard_granularity_bytes = 0` in the client ceph conf and all discards will be applied to the rbd image. The fix should hopefully be backported to stable releases. Thanks for review by the ceph-team on this. If anyone could confirm that this indeed solves the problem, please let me know anyhow. Regards Josef On Thu, Dec 8, 2022 at 11:15 AM Josef Johansson <josef86(a)gmail.com> wrote: > > Hi, > > Running a simple > `echo 1>a;sync;rm a;sync;fstrim --all` > Triggers the problem. No need to have the mount point mounted with discard. > > On Thu, Dec 8, 2022 at 12:33 AM Josef Johansson <josef86(a)gmail.com> wrote: > > > > Hi, > > > > I've updated https://tracker.ceph.com/issues/57396 with some more > > info, it seems that disabling discard within a guest solves the > > problem (or switching from virtio-scsi-single to virtio-blk in older > > kernels). I'm testing two different VMs on the same hypervisor with > > identical configs, one works the other doesn't. > > > > Not sure what to make of it, seems that the kernel around 4.18+ are > > sending a weird discard? > > > > On Tue, Aug 30, 2022 at 8:43 AM Josef Johansson <josef86(a)gmail.com> wrote: > > > > > > Hi, > > > > > > There's nothing special in the cluster when it stops replaying. It > > > seems that a journal entry that the local replayer doesn't handle and > > > just stops. Since it's the local replayer that stops there's no logs > > > in rbd-mirror. The odd part is that rbd-mirror handles this totally > > > fine and is the one syncing correctly. > > > > > > What's worse is that this is reported as HEALTHY in status > > > information, even though when restarting that VM it will stall until > > > replaying is complete. The replay function inside rbd client seems to > > > be fine handling the journal, but only on start of the vm. I will try > > > to get a ticket open on tracker.ceph.com as soon as my account is > > > approved. > > > > > > I have tried to see what component is responsible for local replay but > > > I have not been successful yet. > > > > > > Thanks for answering :) > > > > > > On Mon, Aug 22, 2022 at 11:05 AM Eugen Block <eblock(a)nde.ag> wrote: > > > > > > > > Hi, > > > > > > > > IIRC the rbd mirror journals will grow if the sync stops to work, > > > > which seems to be the case here. Does the primary cluster experience > > > > any high load when the replay stops? How is the connection between the > > > > two sites and is the link saturated? Does the rbd-mirror log reveal > > > > anything useful (maybe also in debug mode)? > > > > > > > > Regards, > > > > Eugen > > > > > > > > Zitat von Josef Johansson <josef(a)oderland.se>: > > > > > > > > > Hi, > > > > > > > > > > I'm running ceph octopus 15.2.16 and I'm trying out two way mirroring. > > > > > > > > > > Everything seems to running fine except sometimes when the replay > > > > > stops at the primary clusters. > > > > > > > > > > This means that VMs will not start properly until all journal > > > > > entries are replayed, but also that the journal grows by time. > > > > > > > > > > I am trying to find out why this occurs, and where to look for more > > > > > information. > > > > > > > > > > I am currently using rbd --pool <pool> --image <image> journal > > > > > status to see if the clients are in sync or not. > > > > > > > > > > Example output when things went sideways > > > > > > > > > > minimum_set: 0 > > > > > active_set: 2 > > > > > registered clients: > > > > > [id=, commit_position=[positions=[[object_number=0, tag_tid=1, > > > > > entry_tid=4592], [object_number=3, tag_tid=1, entry_tid=4591], > > > > > [object_number=2, tag_tid=1, entry_tid=4590], [object_number=1, > > > > > tag_tid=1, entry_tid=4589]]], state=connected] > > > > > [id=bdde9b90-df26-4e3d-84b3-66605dc45608, > > > > > commit_position=[positions=[[object_number=5, tag_tid=1, > > > > > entry_tid=19913], [object_number=4, tag_tid=1, entry_tid=19912], > > > > > [object_number=7, tag_tid=1, entry_tid=19911], [object_number=6, > > > > > tag_tid=1, entry_tid=19910]]], state=disconnected] > > > > > > > > > > Right now I'm trying to catch it red handed in the primary osd logs. > > > > > But I'm not even sure if that's the process that is replaying the > > > > > journal.. > > > > > > > > > > Regards > > > > > Josef > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Octopus RGW large omaps in usage
by Boris Behrens 10 Jan '23

10 Jan '23

Hi, I am currently trying to figure out how to resolve the "large objects found in pool 'rgw.usage'" error. In the past I trimmed the usage log, but now I am at the point that I need to trim it down to two weeks. I checked and amount of omapkeys and the distribution is quite off: # for OBJECT in `rados -p rgw.usage ls`; do rados -p eu-central-1.rgw.usage listomapkeys ${OBJECT} | wc -l done 86968 144388 6188 87854 46652 194788 46234 9622 45768 28376 104348 10018 11112 34374 44744 40638 93664 35476 107794 18020 7172 17836 37344 73496 15572 31570 149352 740 113566 35292 5318 442176 Maybe it would be an option to increase this value?rgw usage max user shards -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Re: Serious cluster issue - Incomplete PGs
by Deep Dish 10 Jan '23

10 Jan '23

Thanks for the insight Eugen. Here's what basically happened: - Upgrade from Nautilus to Quincy via migration to new cluster on temp hardware; - Data from Nautilus migrated successfully to older / lab-type equipment running Quincy; - Nautilus Hardware rebuilt for Quincy, data migrated back; - As data was migrating we set the older notes to maintenance mode and started to drain them; - After several days many OSDs were showing as spinning in "deleting" status on portal and we were marked OUT; - This point we made the incorrect assumption those OSDs were no longer required and proceeded to remove those nodes / OSDs. I understand Incomplete pages are basically lost. And it's likely a lengthy task to attempt to salvage data. Backups will be challenging. I honestly didn't anticipate this kind of failure with ceph to be possible, we've been using it for several years now and were encouraged by orchestrator and performance improvements in the 17 code branch. The fact is of the Incomplete pages that have object counts > 0, there's about 644 GB of data that's tied up in this mess. There are other incomplete PGs with object = 0 which I understand can be manually marked as complete. The cluster has a data usage of 61 TiB. Of this I can categorize about 14TB of critical data, 40 TB of data that is of medium / high importance. There's 14TB in RBD images that would be critical on an EC pool there are other images, however of lower importance at this point; There's also about a 20TB CephFS file system of lower data importance as well. Question - Can you kindly point me to procedures for: - Identifying the pools / images / files that are affected by incomplete pages; - Extracting and reconstructing data for RBD images (these images are XFS formatted filesystems); - Extracting and reconstructing data for CephFS Files not affected by incomplete PGs. Much appreciated. ------------------------------ Date: Mon, 09 Jan 2023 10:12:49 +0000 From: Eugen Block <eblock(a)nde.ag> Subject: [ceph-users] Re: Serious cluster issue - Incomplete PGs To: ceph-users(a)ceph.io Message-ID: <20230109101249.Horde.hAHCWQijFMYLNdX8a2YQDVV(a)webmail.nde.ag> Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Hi, can you clarify what exactly you did to get into this situation? What about the undersized PGs, any chance to bring those OSDs back online? Regarding the incomplete PGs I'm not sure there's much you can do if the OSDs are lost. To me it reads like you may have destroyed/recreated more OSDs than you should have, just recreating OSDs with the same IDs is not sufficient if you destroyed too many chunks. Each OSD only contains a chunk of the PG due to the erasure coding. I'm afraid those objects are lost and you would have to restore from backup. To get the cluster into a healthy state again there a couple of threads, e. g. [1], but recovering the lost chunks from ceph will probably not work. Regards, Eugen [1] https://www.mail-archive.com/ceph-users@ceph.io/msg14757.html Zitat von Deep Dish <deeepdish(a)gmail.com>: > Hello. I really screwed up my ceph cluster. Hoping to get data off it > so I can rebuild it. > > In summary, too many changes too quickly caused the cluster to develop > incomplete pgs. Some PGS were reporting that OSDs were to be probes. > I've created those OSD IDs (empty), however this wouldn't clear > incompletes. Incompletes are part of EC pools. Running 17.2.5. > > This is the overall state: > > cluster: > > id: 49057622-69fc-11ed-b46e-d5acdedaae33 > > health: HEALTH_WARN > > Failed to apply 1 service(s): osd.dashboard-admin-1669078094056 > > 1 hosts fail cephadm check > > cephadm background work is paused > > Reduced data availability: 28 pgs inactive, 28 pgs incomplete > > Degraded data redundancy: 55 pgs undersized > > 2 slow ops, oldest one blocked for 4449 sec, daemons > [osd.25,osd.50,osd.51] have slow ops. > > > > These are PGs that are incomplete that HAVE DATA (Objects > 0) [ via ceph > pg ls incomplete ]: > > 2.35 23199 0 0 0 95980273664 0 > 0 2477 incomplete 10s 2104'46277 28260:686871 > [44,4,37,3,40,32]p44 [44,4,37,3,40,32]p44 > 2023-01-03T03:54:47.821280+0000 2022-12-29T18:53:09.287203+0000 > 14 queued for deep scrub > 2.53 22821 0 0 0 94401175552 0 > 0 2745 remapped+incomplete 10s 2104'45845 28260:565267 > [60,48,52,65,67,7]p60 [60]p60 > 2023-01-03T10:18:13.388383+0000 2023-01-03T10:18:13.388383+0000 > 408 queued for scrub > 2.9f 22858 0 0 0 94555983872 0 > 0 2736 remapped+incomplete 10s 2104'45636 28260:759872 > [56,59,3,57,5,32]p56 [56]p56 > 2023-01-03T10:55:49.848693+0000 2023-01-03T10:55:49.848693+0000 > 376 queued for scrub > 2.be 22870 0 0 0 94429110272 0 > 0 2661 remapped+incomplete 10s 2104'45561 28260:813759 > [41,31,37,9,7,69]p41 [41]p41 > 2023-01-03T14:02:15.790077+0000 2023-01-03T14:02:15.790077+0000 > 360 queued for scrub > 2.e4 22953 0 0 0 94912278528 0 > 0 2648 remapped+incomplete 20m 2104'46048 28259:732896 > [37,46,33,4,48,49]p37 [37]p37 > 2023-01-02T18:38:46.268723+0000 2022-12-29T18:05:47.431468+0000 > 18 queued for deep scrub > 17.78 20169 0 0 0 84517834400 0 > 0 2198 remapped+incomplete 10s 3735'53405 28260:1243673 > [4,37,2,36,66,0]p4 [41]p41 > 2023-01-03T14:21:41.563424+0000 2023-01-03T14:21:41.563424+0000 > 348 queued for scrub > 17.d8 20328 0 0 0 85196053130 0 > 0 1852 remapped+incomplete 10s 3735'54458 28260:1309564 > [38,65,61,37,58,39]p38 [53]p53 > 2023-01-02T18:32:35.371071+0000 2022-12-28T19:08:29.492244+0000 > 21 queued for deep scrub > > At present I'm unable to reliably access my data due to incomplete pages > above. I'll post whatever outputs requested (won't post now as it can be > rather verbose). Is there hope?

2 1