4 Sep
2019
4 Sep
'19
9:48 p.m.
Our test cluster is seeing a problem where peering is going incredibly slow shortly after
upgrading it to Nautilus (14.2.2) from Luminous (12.2.12).
From what I can tell it seems to be caused by "wait for new map" taking a long
time. When looking at dump_historic_slow_ops on pretty much any OSD I see stuff like
this:
# ceph daemon osd.112 dump_historic_slow_ops
[...snip...]
{
"description": "osd_pg_create(e180614 287.4b:177739
287.75:177739 287.1c3:177739 287.1cf:177739 287.1e1:177739 287.2dd:177739 287.2fc:177739
287.342:177739 287.382:177739)",
"initiated_at": "2019-09-03 15:12:41.366514",
"age": 4800.8847047119998,
"duration": 4780.0579745630002,
"type_data": {
"flag_point": "started",
"events": [
{
"time": "2019-09-03 15:12:41.366514",
"event": "initiated"
},
{
"time": "2019-09-03 15:12:41.366514",
"event": "header_read"
},
{
"time": "2019-09-03 15:12:41.366501",
"event": "throttled"
},
{
"time": "2019-09-03 15:12:41.366547",
"event": "all_read"
},
{
"time": "2019-09-03 15:39:03.379456",
"event": "dispatched"
},
{
"time": "2019-09-03 15:39:03.379477",
"event": "wait for new map"
},
{
"time": "2019-09-03 15:39:03.522376",
"event": "wait for new map"
},
{
"time": "2019-09-03 15:53:55.912499",
"event": "wait for new map"
},
{
"time": "2019-09-03 15:59:37.909063",
"event": "wait for new map"
},
{
"time": "2019-09-03 16:00:43.356023",
"event": "wait for new map"
},
{
"time": "2019-09-03 16:20:50.575498",
"event": "wait for new map"
},
{
"time": "2019-09-03 16:31:48.689415",
"event": "started"
},
{
"time": "2019-09-03 16:32:21.424489",
"event": "done"
}
]
}
It always seems to be in osd_pg_create() with multiple "wait for new map"
messages before it finally does something. What could be causing it so long to get the
OSD map? The mons don't appear to be overloaded in any way.
Thanks,
Bryan