Date: Mon, 5 Feb 2024 11:10:22 +0100 From: Jan Marek To: ceph-users@ceph.io Subject: RoCE? Message-ID: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="ZY1AzisMP0c9eJx3" Content-Disposition: inline --ZY1AzisMP0c9eJx3 Content-Type: multipart/mixed; boundary="abJBg20R28c38IZC" Content-Disposition: inline --abJBg20R28c38IZC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, we are configure new ceph cluster with Mellanox 2x100Gbps cards. We bond this two ports to MLAG bond0 interface. In the async+posix mode everythink is OK, cluster is in the HELTH_OK state. CEPH version is 18.2.1. Then we tried to configure RoCE for cluster part of network, but without success. Our ceph config dump (only relevant config): global advanced ms_async_rdma_device_name m= lx5_bond_0 = * global advanced ms_async_rdma_gid_idx 3 global host:ceph1-nvme advanced ms_async_rdma_local_gid 0= 000:0000:0000:0000:0000:ffff:a0d9:05d8 = * global host:ceph2-nvme advanced ms_async_rdma_local_gid 0= 000:0000:0000:0000:0000:ffff:a0d9:05d7 = * global host:ceph3-nvme advanced ms_async_rdma_local_gid 0= 000:0000:0000:0000:0000:ffff:a0d9:05d6 = * global advanced ms_async_rdma_roce_ver 2 global advanced ms_async_rdma_type r= dma = * global advanced ms_cluster_type a= sync+rdma = * global advanced ms_public_type a= sync+posix = * On the ceph1-nvme there is this show_gids.sh list: # ./show_gids.sh DEV PORT INDEX GID IPv4 = VER DEV --- ---- ----- --- -----------= - --- --- mlx5_bond_0 1 0 fe80:0000:0000:0000:0e42:a1ff:fe93:b004 = v1 bond0 mlx5_bond_0 1 1 fe80:0000:0000:0000:0e42:a1ff:fe93:b004 = v2 bond0 mlx5_bond_0 1 2 0000:0000:0000:0000:0000:ffff:a0d9:05d8 160= =2E217.5.216 v1 bond0 mlx5_bond_0 1 3 0000:0000:0000:0000:0000:ffff:a0d9:05d8 160= =2E217.5.216 v2 bond0 n_gids_found=3D4 I have set this line in /etc/security/limits.conf: * hard memlock unlimited But when I tried to restart ceph.target, OSD nodes didn't start with this errors, see attachment. Mellanox drivers are from Debian bookworm kernel. Is there somethink missing in config, or some errors? When I change ms_cluster_type to async+posix and restart ceph.target, cluster converged to HEALTH_OK state... Thanks for advices... Sincerely Jan Marek --=20 Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html --abJBg20R28c38IZC Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="ceph-osd-IB-error.txt" Content-Transfer-Encoding: quoted-printable 2024-02-05T09:56:50.249344+01:00 ceph3-nvme ceph-osd[21139]: auth: KeyRing:= :load: loaded key file /var/lib/ceph/osd/ceph-2/keyring 2024-02-05T09:56:50.249362+01:00 ceph3-nvme ceph-osd[21139]: auth: KeyRing:= :load: loaded key file /var/lib/ceph/osd/ceph-2/keyring 2024-02-05T09:56:50.249377+01:00 ceph3-nvme ceph-osd[21139]: asok(0x5595929= 78000) register_command rotate-key hook 0x7fffcc26d398 2024-02-05T09:56:50.249391+01:00 ceph3-nvme ceph-osd[21139]: log_channel(cl= uster) update_config to_monitors: true to_syslog: false syslog_facility: p= rio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) 2024-02-05T09:56:50.249409+01:00 ceph3-nvme ceph-osd[21139]: osd.2 109 log_= to_monitors true 2024-02-05T09:56:50.249424+01:00 ceph3-nvme ceph-osd[21139]: /home/jenkins-= build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABL= E_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.1/rpm/el8/BU= ILD/ceph-18.2.1/src/msg/async/rdma/Infiniband.cc: In function 'void Infinib= and::init()' thread 7fb3bb042700 time 2024-02-05T08:56:50.142198+0000#012/h= ome/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86= _64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.= 1/rpm/el8/BUILD/ceph-18.2.1/src/msg/async/rdma/Infiniband.cc: 1061: FAILED = ceph_assert(device)#012#012 ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6= dd990c54b29c76) reef (stable)#012 1: (ceph::__ceph_assert_fail(char const*,= char const*, int, char const*)+0x135) [0x55958fb905b3]#012 2: /usr/bin/cep= h-osd(+0x62d779) [0x55958fb90779]#012 3: (Infiniband::init()+0x95b) [0x5595= 9077df0b]#012 4: (RDMAWorker::listen(entity_addr_t&, unsigned int, SocketOp= tions const&, ServerSocket*)+0x30) [0x55959053bba0]#012 5: /usr/bin/ceph-os= d(+0xfbb03f) [0x55959051e03f]#012 6: (EventCenter::process_events(unsigned = int, std::chrono::duration >*)+0= xa64) [0x55959052f4c4]#012 7: /usr/bin/ceph-osd(+0xfd12c6) [0x5595905342c6]= #012 8: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb3c1adcb23]#012 9: /lib64/libp= thread.so.0(+0x81ca) [0x7fb3c248e1ca]#012 10: clone() 2024-02-05T09:56:50.249447+01:00 ceph3-nvme ceph-osd[21139]: *** Caught sig= nal (Aborted) **#012 in thread 7fb3bb042700 thread_name:msgr-worker-0#012#0= 12 ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (sta= ble)#012 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb3c2498d20]#012 2: gsigna= l()#012 3: abort()#012 4: (ceph::__ceph_assert_fail(char const*, char const= *, int, char const*)+0x18f) [0x55958fb9060d]#012 5: /usr/bin/ceph-osd(+0x62= d779) [0x55958fb90779]#012 6: (Infiniband::init()+0x95b) [0x55959077df0b]#0= 12 7: (RDMAWorker::listen(entity_addr_t&, unsigned int, SocketOptions const= &, ServerSocket*)+0x30) [0x55959053bba0]#012 8: /usr/bin/ceph-osd(+0xfbb03f= ) [0x55959051e03f]#012 9: (EventCenter::process_events(unsigned int, std::c= hrono::duration >*)+0xa64) [0x55= 959052f4c4]#012 10: /usr/bin/ceph-osd(+0xfd12c6) [0x5595905342c6]#012 11: /= lib64/libstdc++.so.6(+0xc2b23) [0x7fb3c1adcb23]#012 12: /lib64/libpthread.s= o.0(+0x81ca) [0x7fb3c248e1ca]#012 13: clone()#012 NOTE: a copy of the execu= table, or `objdump -rdS ` is needed to interpret this. 2024-02-05T09:56:50.249475+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: -2717> 2024-02-05T08:56:50.132+0000 7fb3c49da640 = -1 osd.2 109 log_to_monitors true 2024-02-05T09:56:50.249503+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: -2716> 2024-02-05T08:56:50.140+0000 7fb3bb042700 = -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARC= H/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/= 18.2.1/rpm/el8/BUILD/ceph-18.2.1/src/msg/async/rdma/Infiniband.cc: In funct= ion 'void Infiniband::init()' thread 7fb3bb042700 time 2024-02-05T08:56:50.= 142198+0000 2024-02-05T09:56:50.249530+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: /home/jenkins-build/build/workspace/ceph-build/ARC= H/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_= SIZE/gigantic/release/18.2.1/rpm/el8/BUILD/ceph-18.2.1/src/msg/async/rdma/I= nfiniband.cc: 1061: FAILED ceph_assert(device) 2024-02-05T09:56:50.249551+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]:=20 2024-02-05T09:56:50.249571+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd= 990c54b29c76) reef (stable) 2024-02-05T09:56:50.249593+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 1: (ceph::__ceph_assert_fail(char const*, char co= nst*, int, char const*)+0x135) [0x55958fb905b3] 2024-02-05T09:56:50.249614+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 2: /usr/bin/ceph-osd(+0x62d779) [0x55958fb90779] 2024-02-05T09:56:50.249634+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 3: (Infiniband::init()+0x95b) [0x55959077df0b] 2024-02-05T09:56:50.249656+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 4: (RDMAWorker::listen(entity_addr_t&, unsigned i= nt, SocketOptions const&, ServerSocket*)+0x30) [0x55959053bba0] 2024-02-05T09:56:50.249678+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 5: /usr/bin/ceph-osd(+0xfbb03f) [0x55959051e03f] 2024-02-05T09:56:50.249699+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 6: (EventCenter::process_events(unsigned int, std= ::chrono::duration >*)+0xa64) [0= x55959052f4c4] 2024-02-05T09:56:50.249721+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 7: /usr/bin/ceph-osd(+0xfd12c6) [0x5595905342c6] 2024-02-05T09:56:50.249741+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 8: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb3c1adcb2= 3] 2024-02-05T09:56:50.249761+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fb3c248e1c= a] 2024-02-05T09:56:50.249784+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 10: clone() 2024-02-05T09:56:50.249804+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]:=20 2024-02-05T09:56:50.249824+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: -2715> 2024-02-05T08:56:50.168+0000 7fb3bb042700 = -1 *** Caught signal (Aborted) ** 2024-02-05T09:56:50.249843+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: in thread 7fb3bb042700 thread_name:msgr-worker-0 2024-02-05T09:56:50.249862+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]:=20 2024-02-05T09:56:50.249881+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd= 990c54b29c76) reef (stable) 2024-02-05T09:56:50.249901+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb3c2498d= 20] 2024-02-05T09:56:50.249920+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 2: gsignal() 2024-02-05T09:56:50.249940+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 3: abort() 2024-02-05T09:56:50.249959+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 4: (ceph::__ceph_assert_fail(char const*, char co= nst*, int, char const*)+0x18f) [0x55958fb9060d] 2024-02-05T09:56:50.249982+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 5: /usr/bin/ceph-osd(+0x62d779) [0x55958fb90779] 2024-02-05T09:56:50.250002+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 6: (Infiniband::init()+0x95b) [0x55959077df0b] 2024-02-05T09:56:50.250021+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 7: (RDMAWorker::listen(entity_addr_t&, unsigned i= nt, SocketOptions const&, ServerSocket*)+0x30) [0x55959053bba0] 2024-02-05T09:56:50.250042+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 8: /usr/bin/ceph-osd(+0xfbb03f) [0x55959051e03f] 2024-02-05T09:56:50.250066+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 9: (EventCenter::process_events(unsigned int, std= ::chrono::duration >*)+0xa64) [0= x55959052f4c4] 2024-02-05T09:56:50.250090+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 10: /usr/bin/ceph-osd(+0xfd12c6) [0x5595905342c6] 2024-02-05T09:56:50.250110+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 11: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb3c1adcb= 23] 2024-02-05T09:56:50.250132+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fb3c248e1= ca] 2024-02-05T09:56:50.250154+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: 13: clone()=20 2024-02-05T09:56:50.250174+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]: NOTE: a copy of the executable, or `objdump -rdS = ` is needed to interpret this. 2024-02-05T09:56:50.250196+01:00 ceph3-nvme ceph-87483e28-c19a-11ee-90ed-0c= 42a193b004-osd-2[21135]:=20 2024-02-05T09:56:50.332927+01:00 ceph3-nvme podman[21994]: 2024-02-05 09:56= :50.332706257 +0100 CET m=3D+0.015245010 container died 7b35d0bcf4e2c6af121= e5973c4fec96940f5626f3eb7530d560d83bc14a7fea4 (image=3Dquay.io/ceph/ceph@sh= a256:a4e86c750cc11a8c93453ef5682acfa543e3ca08410efefa30f520b54f41831f, name= =3Dceph-87483e28-c19a-11ee-90ed-0c42a193b004-osd-2, org.label-schema.vendor= =3DCentOS, CEPH_POINT_RELEASE=3D-18.2.1, GIT_CLEAN=3DTrue, maintainer=3DGui= llaume Abrioux , org.label-schema.name=3DCentOS Stream= 8 Base Image, GIT_COMMIT=3D4e397bbc7ff93e76025ef390087dfcea05ef676e, org.l= abel-schema.schema-version=3D1.0, org.label-schema.build-date=3D20240102, o= rg.label-schema.license=3DGPLv2, GIT_BRANCH=3DHEAD, RELEASE=3DHEAD, io.buil= dah.version=3D1.29.1, GIT_REPO=3Dhttps://github.com/ceph/ceph-container.git= , ceph=3DTrue) 2024-02-05T09:56:50.337613+01:00 ceph3-nvme systemd[1]: var-lib-containers-= storage-overlay-3df2065cca86c325ca5c08e2c4dd88773c177351b5ff65b21e53ace6e5b= 2fecb-merged.mount: Deactivated successfully. 2024-02-05T09:56:50.341278+01:00 ceph3-nvme podman[21994]: 2024-02-05 09:56= :50.341208686 +0100 CET m=3D+0.023747429 container remove 7b35d0bcf4e2c6af1= 21e5973c4fec96940f5626f3eb7530d560d83bc14a7fea4 (image=3Dquay.io/ceph/ceph@= sha256:a4e86c750cc11a8c93453ef5682acfa543e3ca08410efefa30f520b54f41831f, na= me=3Dceph-87483e28-c19a-11ee-90ed-0c42a193b004-osd-2, ceph=3DTrue, org.labe= l-schema.schema-version=3D1.0, io.buildah.version=3D1.29.1, GIT_REPO=3Dhttp= s://github.com/ceph/ceph-container.git, GIT_BRANCH=3DHEAD, GIT_COMMIT=3D4e3= 97bbc7ff93e76025ef390087dfcea05ef676e, maintainer=3DGuillaume Abrioux , org.label-schema.license=3DGPLv2, org.label-schema.vendor= =3DCentOS, org.label-schema.build-date=3D20240102, org.label-schema.name=3D= CentOS Stream 8 Base Image, GIT_CLEAN=3DTrue, RELEASE=3DHEAD, CEPH_POINT_RE= LEASE=3D-18.2.1) 2024-02-05T09:56:50.343181+01:00 ceph3-nvme systemd[1]: ceph-87483e28-c19a-= 11ee-90ed-0c42a193b004@osd.2.service: Main process exited, code=3Dexited, s= tatus=3D139/n/a --abJBg20R28c38IZC-- --ZY1AzisMP0c9eJx3 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEABYIAB0WIQQUTxPtzAjqwmJlmbXiFFMStj9tyQUCZcC0DQAKCRDiFFMStj9t yXo2AQC7F3Rt9Zmof4g4OlG1N6jrpWTAcJCT69zFZOPPQFqNVgD8DQ6sRcCpw9D4 wqUp0IpL4D/Y0unSHr+7f0FYxZrgfwc= =Qb0C -----END PGP SIGNATURE----- --ZY1AzisMP0c9eJx3--