We are facing the similar issue where we are seeing "libceph: wrong peer, want
<ip:port/obj>, got <ip:port/obj>" in our dmesg as well.
Servers are running Ubuntu 20.04.6 kernel verison: 5.15.0-79-generic
K8s: 1.27.4
containerd:1.6.22
rook: 1.12.1
Ceph: 18.2.0
The rook and ceph versions were recently upgraded from 1.11.9 and 17.2.6 respectively -
these messages we not seen before.
Here are some related dmesg logs from one of our server where we are seeing OSD restarts
for your reference:
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::nn]:6801/3310605789, got
(1)[::nn]:6801/3848687189
[Sun Sep 17 10:21:31 2023] libceph: osd2 (1)[::nn]:6801 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::mm]:6801/480442735, got
(1)[::mm]:6801/261725973
[Sun Sep 17 10:21:31 2023] libceph: osd0 (1)[::mm]:6801 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::yy]:6841/3558675245, got
(1)[::yy]:6841/392097708
[Sun Sep 17 10:21:31 2023] libceph: osd1 (1)[::yy]:6841 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::mm]:6801/3886522490, got
(1)[::mm]:6801/261725973
[Sun Sep 17 10:21:31 2023] libceph: osd0 (1)[::mm]:6801 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::nn]:6801/1808088144, got
(1)[::nn]:6801/3848687189
[Sun Sep 17 10:21:31 2023] libceph: osd2 (1)[::nn]:6801 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::mm]:6801/2444743718, got
(1)[::mm]:6801/261725973
[Sun Sep 17 10:21:31 2023] libceph: osd0 (1)[::mm]:6801 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::yy]:6841/3558675245, got
(1)[::yy]:6841/392097708
[Sun Sep 17 10:21:31 2023] libceph: osd1 (1)[::yy]:6841 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::nn]:6801/927670669, got
(1)[::nn]:6801/3848687189
[Sun Sep 17 10:21:31 2023] libceph: osd2 (1)[::nn]:6801 wrong peer at address
[Sun Sep 17 10:21:31 2023] libceph: wrong peer, want (1)[::mm]:6801/799469619, got
(1)[::mm]:6801/261725973
[Sun Sep 17 10:21:31 2023] libceph: osd0 (1)[::mm]:6801 wrong peer at address
[Sun Sep 17 10:21:32 2023] libceph: wrong peer, want (1)[::yy]:6841/3558675245, got
(1)[::yy]:6841/392097708
[Sun Sep 17 10:21:32 2023] libceph: osd1 (1)[::yy]:6841 wrong peer at address
[Sun Sep 17 10:21:32 2023] libceph: wrong peer, want (1)[::nn]:6801/927670669, got
(1)[::nn]:6801/3848687189
[Sun Sep 17 10:21:32 2023] libceph: osd2 (1)[::nn]:6801 wrong peer at address
[Sun Sep 17 10:21:32 2023] libceph: wrong peer, want (1)[::mm]:6801/799469619, got
(1)[::mm]:6801/261725973
[Sun Sep 17 10:21:32 2023] libceph: osd0 (1)[::mm]:6801 wrong peer at address
[Sun Sep 17 10:24:01 2023] libceph: wrong peer, want (1)[::yy]:6841/3558675245, got
(1)[::yy]:6841/392097708
[Sun Sep 17 10:24:01 2023] libceph: osd1 (1)[::yy]:6841 wrong peer at address
[Sun Sep 17 10:24:01 2023] libceph: wrong peer, want (1)[::yy]:6841/3558675245, got
(1)[::yy]:6841/392097708
[Sun Sep 17 10:24:01 2023] libceph: osd1 (1)[::yy]:6841 wrong peer at address
[Sun Sep 17 10:24:01 2023] libceph: wrong peer, want (1)[::yy]:6841/3558675245, got
(1)[::yy]:6841/392097708
[Sun Sep 17 10:24:01 2023] libceph: osd1 (1)[::yy]:6841 wrong peer at address
[Sun Sep 17 10:24:01 2023] libceph: wrong peer, want (1)[::mm]:6801/799469619, got
(1)[::mm]:6801/261725973
[Sun Sep 17 10:24:01 2023] libceph: osd0 (1)[::mm]:6801 wrong peer at address
Would appreciate some help or insights in resolving the issue. Please let us know if you
need any further information. Thanks.