15 Jan
2020
15 Jan
'20
11:34 a.m.
Hello everyone
If ms_type = async + dpdk, Ceph can work in the master version? I have a problem
using dpdk in the master version. Mon cannot be started. Please hep me, thank you very
much!
1、My network card configuration information：
[root@ebs12 dpdk]# python usertools/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:05:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio
unused=ixgbe
0000:05:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio
unused=ixgbe
Network devices using kernel driver
===================================
0000:08:00.0 'I350 Gigabit Network Connection 1521' if=enp8s0f0 drv=igb
unused=igb_uio
0000:08:00.1 'I350 Gigabit Network Connection 1521' if=enp8s0f1 drv=igb
unused=igb_uio
No 'Crypto' devices detected
============================
No 'Eventdev' devices detected
==============================
No 'Mempool' devices detected
=============================
No 'Compress' devices detected
2、Dpdk can receive and contract normally
testpmd> start
io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP allocation
mode: native
Logical Core 1 (socket 0) forwards packets on 2 streams:
RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
io packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=2
port 0: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=256 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=256 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=32
port 1: RX queue number: 1 Tx queue number: 1
Rx offloads=0x0 Tx offloads=0x0
RX queue: 0
RX desc=256 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
RX Offloads=0x0
TX queue: 0
TX desc=256 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX offloads=0x0 - TX RS bit threshold=32
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
---------------------- Forward statistics for port 0 ----------------------
RX-packets: 10 RX-dropped: 0 RX-total: 10
TX-packets: 9 TX-dropped: 0 TX-total: 9
RX-bursts : 10 [100% of 1 pkts]
TX-bursts : 9 [100% of 1 pkts]
----------------------------------------------------------------------------
---------------------- Forward statistics for port 1 ----------------------
RX-packets: 9 RX-dropped: 0 RX-total: 9
TX-packets: 10 TX-dropped: 0 TX-total: 10
RX-bursts : 9 [100% of 1 pkts]
TX-bursts : 10 [100% of 1 pkts]
----------------------------------------------------------------------------
+++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
RX-packets: 19 RX-dropped: 0 RX-total: 19
TX-packets: 19 TX-dropped: 0 TX-total: 19
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
CPU cycles/packet=2538 (total cycles=48235 / total RX packets=19)
Done.
3、When I start Mon, an error will be reported. I set mon addr = 0000:05:00.0 in
ceph.conf
[root@ebs12 build]# ./bin/ceph-mon -i a -c ceph.conf
2020-01-15T10:54:20.795+0800 7fce87f5c540 -1 WARNING: all dangerous and experimental
features are enabled.
2020-01-15T10:54:20.795+0800 7fce87f5c540 -1 WARNING: all dangerous and experimental
features are enabled.
2020-01-15T10:54:20.806+0800 7fce87f5c540 -1 WARNING: all dangerous and experimental
features are enabled.
2020-01-15T10:54:20.807+0800 7fce87f5c540 -1 asok(0x7fce8bafa000)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX
domain socket to '/tmp/ceph-asok.EUSssX/mon.a.asok': (2) No such file or
directory
2020-01-15T10:54:20.972+0800 7fce87f5c540 -1 WARNING: invalid 'mon addr' config
option
continuing with monmap configuration
EAL: Detected 32 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte_mon.a/mp_socket
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: Error - exiting with code: 1
Cause: No Ethernet ports - bye
/home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: In function
'ceph::mutex_debug_detail::mutex_debug_impl<<anonymous>
>::~mutex_debug_impl() [with bool Recursive = false]' thread 7fce2c44b700 time
2020-01-15T10:54:21.019921+0800
/home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: 111: FAILED ceph_assert(r == 0)
ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus
(dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa)
[0x7fce7e8b9030]
2: (()+0x15162b2) [0x7fce7e8b92b2]
3: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34)
[0x7fce7e763ab2]
4: (()+0x39c99) [0x7fce7a002c99]
5: (()+0x39ce7) [0x7fce7a002ce7]
6: (()+0x1e245a4) [0x7fce7f1c75a4]
7: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b)
[0x7fce7f17b809]
8: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a]
9: (()+0x1893ff1) [0x7fce7ec36ff1]
10: (()+0x1895641) [0x7fce7ec38641]
11: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a]
12: (()+0x1d7444f) [0x7fce7f11744f]
13: (()+0x13ab30e) [0x7fce7e74e30e]
14: (()+0x7e65) [0x7fce7b40de65]
15: (clone()+0x6d) [0x7fce7a0c788d]
*** Caught signal (Aborted) **
in thread 7fce2c44b700 thread_name:msgr-worker-0
2020-01-15T10:54:21.043+0800 7fce2c44b700 -1
/home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: In function
'ceph::mutex_debug_detail::mutex_debug_impl<<anonymous>
>::~mutex_debug_impl() [with bool Recursive = false]' thread 7fce2c44b700 time
2020-01-15T10:54:21.019921+0800
/home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: 111: FAILED ceph_assert(r == 0)
ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus
(dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa)
[0x7fce7e8b9030]
2: (()+0x15162b2) [0x7fce7e8b92b2]
3: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34)
[0x7fce7e763ab2]
4: (()+0x39c99) [0x7fce7a002c99]
5: (()+0x39ce7) [0x7fce7a002ce7]
6: (()+0x1e245a4) [0x7fce7f1c75a4]
7: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b)
[0x7fce7f17b809]
8: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a]
9: (()+0x1893ff1) [0x7fce7ec36ff1]
10: (()+0x1895641) [0x7fce7ec38641]
11: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a]
12: (()+0x1d7444f) [0x7fce7f11744f]
13: (()+0x13ab30e) [0x7fce7e74e30e]
14: (()+0x7e65) [0x7fce7b40de65]
15: (clone()+0x6d) [0x7fce7a0c788d]
ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus
(dev)
1: (()+0x1489a10) [0x7fce89412a10]
2: (()+0xf5f0) [0x7fce7b4155f0]
3: (gsignal()+0x37) [0x7fce79fff337]
4: (abort()+0x148) [0x7fce7a000a28]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x379)
[0x7fce7e8b91ff]
6: (()+0x15162b2) [0x7fce7e8b92b2]
7: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34)
[0x7fce7e763ab2]
8: (()+0x39c99) [0x7fce7a002c99]
9: (()+0x39ce7) [0x7fce7a002ce7]
10: (()+0x1e245a4) [0x7fce7f1c75a4]
11: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b)
[0x7fce7f17b809]
12: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a]
13: (()+0x1893ff1) [0x7fce7ec36ff1]
14: (()+0x1895641) [0x7fce7ec38641]
15: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a]
16: (()+0x1d7444f) [0x7fce7f11744f]
17: (()+0x13ab30e) [0x7fce7e74e30e]
18: (()+0x7e65) [0x7fce7b40de65]
19: (clone()+0x6d) [0x7fce7a0c788d]
2020-01-15T10:54:21.067+0800 7fce2c44b700 -1 *** Caught signal (Aborted) **
in thread 7fce2c44b700 thread_name:msgr-worker-0
ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus
(dev)
1: (()+0x1489a10) [0x7fce89412a10]
2: (()+0xf5f0) [0x7fce7b4155f0]
3: (gsignal()+0x37) [0x7fce79fff337]
4: (abort()+0x148) [0x7fce7a000a28]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x379)
[0x7fce7e8b91ff]
6: (()+0x15162b2) [0x7fce7e8b92b2]
7: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34)
[0x7fce7e763ab2]
8: (()+0x39c99) [0x7fce7a002c99]
9: (()+0x39ce7) [0x7fce7a002ce7]
10: (()+0x1e245a4) [0x7fce7f1c75a4]
11: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b)
[0x7fce7f17b809]
12: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a]
13: (()+0x1893ff1) [0x7fce7ec36ff1]
14: (()+0x1895641) [0x7fce7ec38641]
15: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a]
16: (()+0x1d7444f) [0x7fce7f11744f]
17: (()+0x13ab30e) [0x7fce7e74e30e]
18: (()+0x7e65) [0x7fce7b40de65]
19: (clone()+0x6d) [0x7fce7a0c788d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
-250> 2020-01-15T10:54:20.795+0800 7fce87f5c540 -1 WARNING: all dangerous and
experimental features are enabled.
-247> 2020-01-15T10:54:20.795+0800 7fce87f5c540 -1 WARNING: all dangerous and
experimental features are enabled.
-232> 2020-01-15T10:54:20.806+0800 7fce87f5c540 -1 WARNING: all dangerous and
experimental features are enabled.
-229> 2020-01-15T10:54:20.807+0800 7fce87f5c540 -1 asok(0x7fce8bafa000)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX
domain socket to '/tmp/ceph-asok.EUSssX/mon.a.asok': (2) No such file or
directory
-40> 2020-01-15T10:54:20.972+0800 7fce87f5c540 -1 WARNING: invalid 'mon
addr' config option
continuing with monmap configuration
-1> 2020-01-15T10:54:21.043+0800 7fce2c44b700 -1
/home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: In function
'ceph::mutex_debug_detail::mutex_debug_impl<<anonymous>
>::~mutex_debug_impl() [with bool Recursive = false]' thread 7fce2c44b700 time
2020-01-15T10:54:21.019921+0800
/home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: 111: FAILED ceph_assert(r == 0)
ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus
(dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa)
[0x7fce7e8b9030]
2: (()+0x15162b2) [0x7fce7e8b92b2]
3: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34)
[0x7fce7e763ab2]
4: (()+0x39c99) [0x7fce7a002c99]
5: (()+0x39ce7) [0x7fce7a002ce7]
6: (()+0x1e245a4) [0x7fce7f1c75a4]
7: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b)
[0x7fce7f17b809]
8: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a]
9: (()+0x1893ff1) [0x7fce7ec36ff1]
10: (()+0x1895641) [0x7fce7ec38641]
11: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a]
12: (()+0x1d7444f) [0x7fce7f11744f]
13: (()+0x13ab30e) [0x7fce7e74e30e]
14: (()+0x7e65) [0x7fce7b40de65]
15: (clone()+0x6d) [0x7fce7a0c788d]
0> 2020-01-15T10:54:21.067+0800 7fce2c44b700 -1 *** Caught signal (Aborted) **
in thread 7fce2c44b700 thread_name:msgr-worker-0
ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus
(dev)
1: (()+0x1489a10) [0x7fce89412a10]
2: (()+0xf5f0) [0x7fce7b4155f0]
3: (gsignal()+0x37) [0x7fce79fff337]
4: (abort()+0x148) [0x7fce7a000a28]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x379)
[0x7fce7e8b91ff]
6: (()+0x15162b2) [0x7fce7e8b92b2]
7: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34)
[0x7fce7e763ab2]
8: (()+0x39c99) [0x7fce7a002c99]
9: (()+0x39ce7) [0x7fce7a002ce7]
10: (()+0x1e245a4) [0x7fce7f1c75a4]
11: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b)
[0x7fce7f17b809]
12: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a]
13: (()+0x1893ff1) [0x7fce7ec36ff1]
14: (()+0x1895641) [0x7fce7ec38641]
15: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a]
16: (()+0x1d7444f) [0x7fce7f11744f]
17: (()+0x13ab30e) [0x7fce7e74e30e]
18: (()+0x7e65) [0x7fce7b40de65]
19: (clone()+0x6d) [0x7fce7a0c788d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
-250> 2020-01-15T10:54:20.795+0800 7fce87f5c540 -1 WARNING: all dangerous and
experimental features are enabled.
-247> 2020-01-15T10:54:20.795+0800 7fce87f5c540 -1 WARNING: all dangerous and
experimental features are enabled.
-232> 2020-01-15T10:54:20.806+0800 7fce87f5c540 -1 WARNING: all dangerous and
experimental features are enabled.
-229> 2020-01-15T10:54:20.807+0800 7fce87f5c540 -1 asok(0x7fce8bafa000)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX
domain socket to '/tmp/ceph-asok.EUSssX/mon.a.asok': (2) No such file or
directory
-40> 2020-01-15T10:54:20.972+0800 7fce87f5c540 -1 WARNING: invalid 'mon
addr' config option
continuing with monmap configuration
-1> 2020-01-15T10:54:21.043+0800 7fce2c44b700 -1
/home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: In function
'ceph::mutex_debug_detail::mutex_debug_impl<<anonymous>
>::~mutex_debug_impl() [with bool Recursive = false]' thread 7fce2c44b700 time
2020-01-15T10:54:21.019921+0800
/home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: 111: FAILED ceph_assert(r == 0)
ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus
(dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa)
[0x7fce7e8b9030]
2: (()+0x15162b2) [0x7fce7e8b92b2]
3: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34)
[0x7fce7e763ab2]
4: (()+0x39c99) [0x7fce7a002c99]
5: (()+0x39ce7) [0x7fce7a002ce7]
6: (()+0x1e245a4) [0x7fce7f1c75a4]
7: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b)
[0x7fce7f17b809]
8: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a]
9: (()+0x1893ff1) [0x7fce7ec36ff1]
10: (()+0x1895641) [0x7fce7ec38641]
11: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a]
12: (()+0x1d7444f) [0x7fce7f11744f]
13: (()+0x13ab30e) [0x7fce7e74e30e]
14: (()+0x7e65) [0x7fce7b40de65]
15: (clone()+0x6d) [0x7fce7a0c788d]
0> 2020-01-15T10:54:21.067+0800 7fce2c44b700 -1 *** Caught signal (Aborted) **
in thread 7fce2c44b700 thread_name:msgr-worker-0
ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus
(dev)
1: (()+0x1489a10) [0x7fce89412a10]
2: (()+0xf5f0) [0x7fce7b4155f0]
3: (gsignal()+0x37) [0x7fce79fff337]
4: (abort()+0x148) [0x7fce7a000a28]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x379)
[0x7fce7e8b91ff]
6: (()+0x15162b2) [0x7fce7e8b92b2]
7: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34)
[0x7fce7e763ab2]
8: (()+0x39c99) [0x7fce7a002c99]
9: (()+0x39ce7) [0x7fce7a002ce7]
10: (()+0x1e245a4) [0x7fce7f1c75a4]
11: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b)
[0x7fce7f17b809]
12: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a]
13: (()+0x1893ff1) [0x7fce7ec36ff1]
14: (()+0x1895641) [0x7fce7ec38641]
15: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a]
16: (()+0x1d7444f) [0x7fce7f11744f]
17: (()+0x13ab30e) [0x7fce7e74e30e]
18: (()+0x7e65) [0x7fce7b40de65]
19: (clone()+0x6d) [0x7fce7a0c788d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
云计算产品部 郑印
中移（苏州）软件技术有限公司
中国移动云能力中心
18896726650
zhengyin(a)cmss.chinamobile.com
江苏省苏州市高新区科技城昆仑山路58号 中移软件园 215153