9 Mar
2023
9 Mar
'23
12:49 p.m.
Hi all,
we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what
action to take. We get the error "wrong peer at address" in dmesg and some jobs
on that server seem to get stuck in fs access; log extract below. I found these 2 tracker
items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519,
which don't seem to have fixes.
My questions:
- Is this harmless or does it indicate invalid/corrupted client cache entries?
- How to resolve, ignore, umount+mount or reboot?
Here an extract from the dmesg log, the error has survived a couple of MDS restarts
already:
[Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-1572619386
[Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-1572619386
[Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start
[Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start
[Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success
[Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success
[Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22
[Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed
[Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed
[Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start
[Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start
[Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success
[Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success
[Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22
[Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed
[Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed
[Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[...]
[Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3
[Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
13 Mar
13 Mar
3:44 a.m.
Hi Frank,
BTW, what's your kernel version you were using ? It's a bug and I
haven't ever seen this by using the newer kernel.
You can try to remount the mountpoints and it should work.
Thanks
- Xiubo
On 09/03/2023 17:49, Frank Schilder wrote:
Hi all,
we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what
action to take. We get the error "wrong peer at address" in dmesg and some jobs
on that server seem to get stuck in fs access; log extract below. I found these 2 tracker
items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519,
which don't seem to have fixes.
My questions:
- Is this harmless or does it indicate invalid/corrupted client cache entries?
- How to resolve, ignore, umount+mount or reboot?
Here an extract from the dmesg log, the error has survived a couple of MDS restarts
already:
[Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-1572619386
[Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-1572619386
[Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start
[Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start
[Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success
[Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success
[Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22
[Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed
[Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed
[Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start
[Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start
[Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success
[Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success
[Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22
[Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed
[Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed
[Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[...]
[Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3
[Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
12:10 p.m.
Hi Xiubo,
its a really old kernel version: 3.10.0-957.10.1.el7.x86_64. We plan to upgrade soonish,
but its a major operation. For now we just need a workaround to get the client clean
again. Do you have information about what triggers this bug? Maybe we can avoid the
occurrence.
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Xiubo Li <xiubli(a)redhat.com>
Sent: 13 March 2023 01:44:49
To: Frank Schilder; ceph-users(a)ceph.io
Subject: Re: [ceph-users] libceph: mds1 IP+PORT wrong peer at address
Hi Frank,
BTW, what's your kernel version you were using ? It's a bug and I
haven't ever seen this by using the newer kernel.
You can try to remount the mountpoints and it should work.
Thanks
- Xiubo
On 09/03/2023 17:49, Frank Schilder wrote:
Hi all,
we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what
action to take. We get the error "wrong peer at address" in dmesg and some jobs
on that server seem to get stuck in fs access; log extract below. I found these 2 tracker
items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519,
which don't seem to have fixes.
My questions:
- Is this harmless or does it indicate invalid/corrupted client cache entries?
- How to resolve, ignore, umount+mount or reboot?
Here an extract from the dmesg log, the error has survived a couple of MDS restarts
already:
[Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-1572619386
[Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-1572619386
[Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start
[Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start
[Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success
[Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success
[Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22
[Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed
[Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed
[Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start
[Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start
[Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success
[Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success
[Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22
[Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed
[Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed
[Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[...]
[Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3
[Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
2:44 p.m.
Hi Frank,
I am afraid there is buggy in the code and it's racy when updating the
new mdsmap with the old one. We have several fixes about this as I
remembered.
You can try the newer kernels to see could you reproduce it.
Thanks
- Xiubo
On 13/03/2023 17:10, Frank Schilder wrote:
Hi Xiubo,
its a really old kernel version: 3.10.0-957.10.1.el7.x86_64. We plan to upgrade soonish,
but its a major operation. For now we just need a workaround to get the client clean
again. Do you have information about what triggers this bug? Maybe we can avoid the
occurrence.
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Xiubo Li <xiubli(a)redhat.com>
Sent: 13 March 2023 01:44:49
To: Frank Schilder; ceph-users(a)ceph.io
Subject: Re: [ceph-users] libceph: mds1 IP+PORT wrong peer at address
Hi Frank,
BTW, what's your kernel version you were using ? It's a bug and I
haven't ever seen this by using the newer kernel.
You can try to remount the mountpoints and it should work.
Thanks
- Xiubo
On 09/03/2023 17:49, Frank Schilder wrote:
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
Hi all,
we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what
action to take. We get the error "wrong peer at address" in dmesg and some jobs
on that server seem to get stuck in fs access; log extract below. I found these 2 tracker
items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519,
which don't seem to have fixes.
My questions:
- Is this harmless or does it indicate invalid/corrupted client cache entries?
- How to resolve, ignore, umount+mount or reboot?
Here an extract from the dmesg log, the error has survived a couple of MDS restarts
already:
[Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-1572619386
[Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-1572619386
[Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start
[Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start
[Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success
[Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success
[Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22
[Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state
OPEN)
[Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed
[Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed
[Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start
[Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start
[Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success
[Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success
[Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22
[Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed
[Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed
[Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[...]
[Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3
[Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
[Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got
192.168.32.87:6801/-453143347
[Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Best Regards,
Xiubo Li (李秀波)
Email: xiubli@redhat.com/xiubli@ibm.com
Slack: @Xiubo Li
18 Sep
18 Sep
6:27 p.m.
We are facing the similar issue where we are seeing "libceph: wrong peer, want
<ip:port/obj>, got <ip:port/obj>" in our dmesg as well.
Servers are running Ubuntu 20.04.6 kernel verison: 5.15.0-79-generic
K8s: 1.27.4
containerd:1.6.22
rook: 1.12.1
Ceph: 18.2.0
The rook and ceph versions were recently upgraded from 1.11.9 and 17.2.6 respectively -
these messages we not seen before.
Here are some related dmesg logs from one of our server where we are seeing OSD restarts
for your reference:
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::nn]:6801/3310605789, got
(1)[::nn]:6801/3848687189
[Sun Sep 17 10:21:31 2023] libceph: osd2 (1)[::nn]:6801 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::mm]:6801/480442735, got
(1)[::mm]:6801/261725973
[Sun Sep 17 10:21:31 2023] libceph: osd0 (1)[::mm]:6801 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::yy]:6841/3558675245, got
(1)[::yy]:6841/392097708
[Sun Sep 17 10:21:31 2023] libceph: osd1 (1)[::yy]:6841 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::mm]:6801/3886522490, got
(1)[::mm]:6801/261725973
[Sun Sep 17 10:21:31 2023] libceph: osd0 (1)[::mm]:6801 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::nn]:6801/1808088144, got
(1)[::nn]:6801/3848687189
[Sun Sep 17 10:21:31 2023] libceph: osd2 (1)[::nn]:6801 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::mm]:6801/2444743718, got
(1)[::mm]:6801/261725973
[Sun Sep 17 10:21:31 2023] libceph: osd0 (1)[::mm]:6801 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::yy]:6841/3558675245, got
(1)[::yy]:6841/392097708
[Sun Sep 17 10:21:31 2023] libceph: osd1 (1)[::yy]:6841 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::nn]:6801/927670669, got
(1)[::nn]:6801/3848687189
[Sun Sep 17 10:21:31 2023] libceph: osd2 (1)[::nn]:6801 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::mm]:6801/799469619, got
(1)[::mm]:6801/261725973
[Sun Sep 17 10:21:31 2023] libceph: osd0 (1)[::mm]:6801 wrong peer at address
[Sun Sep 17 10:21:32 2023] libceph: wrong peer, want (1)[::yy]:6841/3558675245, got
(1)[::yy]:6841/392097708
[Sun Sep 17 10:21:32 2023] libceph: osd1 (1)[::yy]:6841 wrong peer at address
[Sun Sep 17 10:21:32 2023] libceph: wrong peer, want (1)[::nn]:6801/927670669, got
(1)[::nn]:6801/3848687189
[Sun Sep 17 10:21:32 2023] libceph: osd2 (1)[::nn]:6801 wrong peer at address
[Sun Sep 17 10:21:32 2023] libceph: wrong peer, want (1)[::mm]:6801/799469619, got
(1)[::mm]:6801/261725973
[Sun Sep 17 10:21:32 2023] libceph: osd0 (1)[::mm]:6801 wrong peer at address
[Sun Sep 17 10:24:01 2023] libceph: wrong peer, want (1)[::yy]:6841/3558675245, got
(1)[::yy]:6841/392097708
[Sun Sep 17 10:24:01 2023] libceph: osd1 (1)[::yy]:6841 wrong peer at address
[Sun Sep 17 10:24:01 2023] libceph: wrong peer, want (1)[::yy]:6841/3558675245, got
(1)[::yy]:6841/392097708
[Sun Sep 17 10:24:01 2023] libceph: osd1 (1)[::yy]:6841 wrong peer at address
[Sun Sep 17 10:24:01 2023] libceph: wrong peer, want (1)[::yy]:6841/3558675245, got
(1)[::yy]:6841/392097708
[Sun Sep 17 10:24:01 2023] libceph: osd1 (1)[::yy]:6841 wrong peer at address
[Sun Sep 17 10:24:01 2023] libceph: wrong peer, want (1)[::mm]:6801/799469619, got
(1)[::mm]:6801/261725973
[Sun Sep 17 10:24:01 2023] libceph: osd0 (1)[::mm]:6801 wrong peer at address
Would appreciate some help or insights in resolving the issue. Please let us know if you
need any further information. Thanks.
19 Sep
19 Sep
6:35 p.m.
Hi,
We are unable to resolve these issues and OSD restarts have made the ceph cluster
unusable. We are wondering if downgrading ceph version from 18.2.0 to 17.2.6. Please let
us know if this is supported and if so, please point me to the procedure to do the same.
Thanks.
20 Sep
20 Sep
10:18 a.m.
Hi, in our case the problem was on the client side. When you write "logs from a
host", do you mean an OSD host or a host where client connections come from? Its not
clear from your problem description *who* is requesting the wrong peer.
The reason for this message is that something tries to talk to an older instance of a
daemon. The number after the / is a nonce that is assigned after each restart to be able
to distinguish different instances of the same service (say, OSD or MDS). Usually, these
get updated on peering.
If its clients that are stuck, you probably need to reboot the client hosts.
Please specify what is trying to reach the outdated OSD instances. Then a relevant
developer is more likely to look at it. Since its not MDS-kclient interaction it might be
useful to open a new case.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: U S <ultrasagenexus(a)gmail.com>
Sent: Tuesday, September 19, 2023 5:35 PM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: libceph: mds1 IP+PORT wrong peer at address
Hi,
We are unable to resolve these issues and OSD restarts have made the ceph cluster
unusable. We are wondering if downgrading ceph version from 18.2.0 to 17.2.6. Please let
us know if this is supported and if so, please point me to the procedure to do the same.
Thanks.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
227
days inactive
422
days old
6 comments
4 participants
participants (4)
-
Frank Schilder -
U S -
ultrasagenexus＠gmail.com
-
Xiubo Li