10 Jan
2023
10 Jan
'23
10:46 a.m.
Hi,
Actually, the test case was even more simple than that. A misaligned
discard (discard_granularity_bytes=4096, offset=0, length=4096+512)
made the journal stop replaying entries. This is now well covered in
tests and example e2e-tests.
The workaround is quite easy, set `rbd_discard_granularity_bytes = 0`
in the client ceph conf and all discards will be applied to the rbd
image. The fix should hopefully be backported to stable releases.
Thanks for review by the ceph-team on this.
If anyone could confirm that this indeed solves the problem, please
let me know anyhow.
Regards
Josef
On Thu, Dec 8, 2022 at 11:15 AM Josef Johansson <josef86(a)gmail.com> wrote:
>
> Hi,
>
> Running a simple
> `echo 1>a;sync;rm a;sync;fstrim --all`
> Triggers the problem. No need to have the mount point mounted with discard.
>
> On Thu, Dec 8, 2022 at 12:33 AM Josef Johansson <josef86(a)gmail.com> wrote:
> >
> > Hi,
> >
> > I've updated https://tracker.ceph.com/issues/57396 with some more
> > info, it seems that disabling discard within a guest solves the
> > problem (or switching from virtio-scsi-single to virtio-blk in older
> > kernels). I'm testing two different VMs on the same hypervisor with
> > identical configs, one works the other doesn't.
> >
> > Not sure what to make of it, seems that the kernel around 4.18+ are
> > sending a weird discard?
> >
> > On Tue, Aug 30, 2022 at 8:43 AM Josef Johansson <josef86(a)gmail.com>
wrote:
> > >
> > > Hi,
> > >
> > > There's nothing special in the cluster when it stops replaying. It
> > > seems that a journal entry that the local replayer doesn't handle and
> > > just stops. Since it's the local replayer that stops there's no
logs
> > > in rbd-mirror. The odd part is that rbd-mirror handles this totally
> > > fine and is the one syncing correctly.
> > >
> > > What's worse is that this is reported as HEALTHY in status
> > > information, even though when restarting that VM it will stall until
> > > replaying is complete. The replay function inside rbd client seems to
> > > be fine handling the journal, but only on start of the vm. I will try
> > > to get a ticket open on tracker.ceph.com as soon as my account is
> > > approved.
> > >
> > > I have tried to see what component is responsible for local replay but
> > > I have not been successful yet.
> > >
> > > Thanks for answering :)
> > >
> > > On Mon, Aug 22, 2022 at 11:05 AM Eugen Block <eblock(a)nde.ag> wrote:
> > > >
> > > > Hi,
> > > >
> > > > IIRC the rbd mirror journals will grow if the sync stops to work,
> > > > which seems to be the case here. Does the primary cluster experience
> > > > any high load when the replay stops? How is the connection between
the
> > > > two sites and is the link saturated? Does the rbd-mirror log reveal
> > > > anything useful (maybe also in debug mode)?
> > > >
> > > > Regards,
> > > > Eugen
> > > >
> > > > Zitat von Josef Johansson <josef(a)oderland.se>se>:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'm running ceph octopus 15.2.16 and I'm trying out two
way mirroring.
> > > > >
> > > > > Everything seems to running fine except sometimes when the
replay
> > > > > stops at the primary clusters.
> > > > >
> > > > > This means that VMs will not start properly until all journal
> > > > > entries are replayed, but also that the journal grows by time.
> > > > >
> > > > > I am trying to find out why this occurs, and where to look for
more
> > > > > information.
> > > > >
> > > > > I am currently using rbd --pool <pool> --image
<image> journal
> > > > > status to see if the clients are in sync or not.
> > > > >
> > > > > Example output when things went sideways
> > > > >
> > > > > minimum_set: 0
> > > > > active_set: 2
> > > > > registered clients:
> > > > > [id=, commit_position=[positions=[[object_number=0, tag_tid=1,
> > > > > entry_tid=4592], [object_number=3, tag_tid=1, entry_tid=4591],
> > > > > [object_number=2, tag_tid=1, entry_tid=4590], [object_number=1,
> > > > > tag_tid=1, entry_tid=4589]]], state=connected]
> > > > > [id=bdde9b90-df26-4e3d-84b3-66605dc45608,
> > > > > commit_position=[positions=[[object_number=5, tag_tid=1,
> > > > > entry_tid=19913], [object_number=4, tag_tid=1, entry_tid=19912],
> > > > > [object_number=7, tag_tid=1, entry_tid=19911], [object_number=6,
> > > > > tag_tid=1, entry_tid=19910]]], state=disconnected]
> > > > >
> > > > > Right now I'm trying to catch it red handed in the primary
osd logs.
> > > > > But I'm not even sure if that's the process that is
replaying the
> > > > > journal..
> > > > >
> > > > > Regards
> > > > > Josef
> > > > > _______________________________________________
> > > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io