12 Dec
2019
12 Dec
'19
11:46 p.m.
On our test cluster after upgrading to 14.2.5 I'm having problems with the mons
pegging a CPU core while moving data around. I'm currently converting the OSDs from
FileStore to BlueStore by marking the OSDs out in multiple nodes, destroying the OSDs, and
then recreating them with ceph-volume lvm batch. This seems too get the ceph-mon process
into a state where it pegs a CPU core on one of the mons:
1764450 ceph 20 0 4802412 2.1g 16980 S 100.0 28.1 4:54.72 ceph-mon
Has anyone else run into this with 14.2.5 yet? I didn't see this problem while the
cluster was running 14.2.4.
Thanks,
Bryan
13 Dec
13 Dec
4:07 p.m.
Adding the dev list since it seems like a bug in 14.2.5.
I was able to capture the output from perf top:
21.58% libceph-common.so.0 [.] ceph::buffer::v14_2_0::list::append
20.90% libstdc++.so.6.0.19 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >
13.25% libceph-common.so.0 [.] ceph::buffer::v14_2_0::list::append
10.11% libstdc++.so.6.0.19 [.] std::istream::sentry::sentry
8.94% libstdc++.so.6.0.19 [.] std::basic_ios<char,
std::char_traits<char> >::clear
3.24% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::ptr::unused_tail_length
1.69% libceph-common.so.0 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >@plt
1.63% libstdc++.so.6.0.19 [.] std::istream::sentry::sentry@plt
1.21% [kernel] [k] __do_softirq
0.77% libpython2.7.so.1.0 [.] PyEval_EvalFrameEx
0.55% [kernel] [k] _raw_spin_unlock_irqrestore
I increased mon debugging to 20 and nothing stuck out to me.
Bryan
On Dec 12, 2019, at 4:46 PM, Bryan Stillwell
<bstillwell(a)godaddy.com> wrote:
On our test cluster after upgrading to 14.2.5 I'm having problems with the mons
pegging a CPU core while moving data around. I'm currently converting the OSDs from
FileStore to BlueStore by marking the OSDs out in multiple nodes, destroying the OSDs, and
then recreating them with ceph-volume lvm batch. This seems too get the ceph-mon process
into a state where it pegs a CPU core on one of the mons:
1764450 ceph 20 0 4802412 2.1g 16980 S 100.0 28.1 4:54.72 ceph-mon
Has anyone else run into this with 14.2.5 yet? I didn't see this problem while the
cluster was running 14.2.4.
Thanks,
Bryan
14 Dec
14 Dec
5:27 p.m.
Bryan,
Were you able to resolve this? If yes, can you please share with the list?
On Fri, Dec 13, 2019 at 10:08 AM Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
Adding the dev list since it seems like a bug in
14.2.5.
I was able to capture the output from perf top:
21.58% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::list::append
20.90% libstdc++.so.6.0.19 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >
13.25% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::list::append
10.11% libstdc++.so.6.0.19 [.]
std::istream::sentry::sentry
8.94% libstdc++.so.6.0.19 [.] std::basic_ios<char,
std::char_traits<char> >::clear
3.24% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::ptr::unused_tail_length
1.69% libceph-common.so.0 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >@plt
1.63% libstdc++.so.6.0.19 [.]
std::istream::sentry::sentry@plt
1.21% [kernel] [k] __do_softirq
0.77% libpython2.7.so.1.0 [.] PyEval_EvalFrameEx
0.55% [kernel] [k] _raw_spin_unlock_irqrestore
I increased mon debugging to 20 and nothing stuck out to me.
Bryan
On Dec 12, 2019, at 4:46 PM, Bryan Stillwell
<bstillwell(a)godaddy.com>
wrote:
On our test cluster after upgrading to 14.2.5 I'm having problems with
the
mons pegging a CPU core while moving data around. I'm currently
converting the OSDs from FileStore to BlueStore by marking the OSDs out in
multiple nodes, destroying the OSDs, and then recreating them with
ceph-volume lvm batch. This seems too get the ceph-mon process into a
state where it pegs a CPU core on one of the mons:
1764450 ceph 20 0 4802412 2.1g 16980 S 100.0 28.1 4:54.72
ceph-mon
Has anyone else run into this with 14.2.5 yet? I didn't see this
problem
while the cluster was running 14.2.4.
Thanks,
Bryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
16 Dec
16 Dec
9:57 p.m.
Sasha,
I was able to get past it by restarting the ceph-mon processes every time it got stuck,
but that's not a very good solution for a production cluster.
Right now I'm trying to narrow down what is causing the problem. Rebuilding the OSDs
with BlueStore doesn't seem to be enough. I believe it could be related to us using
the extra space on the journal device as an SSD-based OSD. During the conversion process
I'm removing this SSD-based OSD (since with BlueStore the omap data is ending up on
the SSD anyways), and I'm suspecting it might be causing this problem.
Bryan
On Dec 14, 2019, at 10:27 AM, Sasha Litvak
<alexander.v.litvak@gmail.com<mailto:alexander.v.litvak@gmail.com>> wrote:
Notice: This email is from an external sender.
Bryan,
Were you able to resolve this? If yes, can you please share with the list?
On Fri, Dec 13, 2019 at 10:08 AM Bryan Stillwell
<bstillwell@godaddy.com<mailto:bstillwell@godaddy.com>> wrote:
Adding the dev list since it seems like a bug in 14.2.5.
I was able to capture the output from perf top:
21.58% libceph-common.so.0 [.] ceph::buffer::v14_2_0::list::append
20.90% libstdc++.so.6.0.19 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >
13.25% libceph-common.so.0 [.] ceph::buffer::v14_2_0::list::append
10.11% libstdc++.so.6.0.19 [.] std::istream::sentry::sentry
8.94% libstdc++.so.6.0.19 [.] std::basic_ios<char,
std::char_traits<char> >::clear
3.24% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::ptr::unused_tail_length
1.69% libceph-common.so.0 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >@plt
1.63% libstdc++.so.6.0.19 [.] std::istream::sentry::sentry@plt
1.21% [kernel] [k] __do_softirq
0.77% libpython2.7.so.1.0 [.] PyEval_EvalFrameEx
0.55% [kernel] [k] _raw_spin_unlock_irqrestore
I increased mon debugging to 20 and nothing stuck out to me.
Bryan
On Dec 12, 2019, at 4:46 PM, Bryan Stillwell
<bstillwell@godaddy.com<mailto:bstillwell@godaddy.com>> wrote:
On our test cluster after upgrading to 14.2.5 I'm having problems with the mons
pegging a CPU core while moving data around. I'm currently converting the OSDs from
FileStore to BlueStore by marking the OSDs out in multiple nodes, destroying the OSDs, and
then recreating them with ceph-volume lvm batch. This seems too get the ceph-mon process
into a state where it pegs a CPU core on one of the mons:
1764450 ceph 20 0 4802412 2.1g 16980 S 100.0 28.1 4:54.72 ceph-mon
Has anyone else run into this with 14.2.5 yet? I didn't see this problem while the
cluster was running 14.2.4.
Thanks,
Bryan
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
17 Dec
17 Dec
12:51 a.m.
Bryan, thank you. We are about to start testing 14.2.2 -> 14.2.5 upgrade,
so folks here are a bit cautious :-) We don't need to convert but may have
to rebuild few disks after an upgrade.
On Mon, Dec 16, 2019 at 3:57 PM Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
Sasha,
I was able to get past it by restarting the ceph-mon processes every time
it got stuck, but that's not a very good solution for a production cluster.
Right now I'm trying to narrow down what is causing the problem.
Rebuilding the OSDs with BlueStore doesn't seem to be enough. I believe it
could be related to us using the extra space on the journal device as an
SSD-based OSD. During the conversion process I'm removing this SSD-based
OSD (since with BlueStore the omap data is ending up on the SSD anyways),
and I'm suspecting it might be causing this problem.
Bryan
On Dec 14, 2019, at 10:27 AM, Sasha Litvak <alexander.v.litvak(a)gmail.com>
wrote:
Notice: This email is from an external sender.
Bryan,
Were you able to resolve this? If yes, can you please share with the list?
On Fri, Dec 13, 2019 at 10:08 AM Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
Adding the dev list since it seems like a bug in
14.2.5.
I was able to capture the output from perf top:
21.58% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::list::append
20.90% libstdc++.so.6.0.19 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >
13.25% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::list::append
10.11% libstdc++.so.6.0.19 [.]
std::istream::sentry::sentry
8.94% libstdc++.so.6.0.19 [.] std::basic_ios<char,
std::char_traits<char> >::clear
3.24% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::ptr::unused_tail_length
1.69% libceph-common.so.0 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >@plt
1.63% libstdc++.so.6.0.19 [.]
std::istream::sentry::sentry@plt
1.21% [kernel] [k] __do_softirq
0.77% libpython2.7.so.1.0 [.] PyEval_EvalFrameEx
0.55% [kernel] [k]
_raw_spin_unlock_irqrestore
I increased mon debugging to 20 and nothing stuck out to me.
Bryan
On Dec 12, 2019, at 4:46 PM, Bryan Stillwell
<bstillwell(a)godaddy.com>
wrote:
On our test cluster after upgrading to 14.2.5 I'm having problems with
the
mons pegging a CPU core while moving data around. I'm currently
converting the OSDs from FileStore to BlueStore by marking the OSDs out in
multiple nodes, destroying the OSDs, and then recreating them with
ceph-volume lvm batch. This seems too get the ceph-mon process into a
state where it pegs a CPU core on one of the mons:
1764450 ceph 20 0 4802412 2.1g 16980 S 100.0 28.1 4:54.72
ceph-mon
Has anyone else run into this with 14.2.5 yet? I didn't see this
problem
while the cluster was running 14.2.4.
Thanks,
Bryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
1:41 a.m.
Hi guys,
I am running red hat ceph (basically luminous - ceph version
12.2.12-48.el7cp (26388d73d88602005946d4381cc5796d42904858)) and am seeing
something similar on our test cluster.
One of the mons is running at around 300% cpu non stop. It doesn't seem to
be the lead mon or one in particular, but the cpu load shifts to another
mon if high load mon is restarted.
I thought it might be related to this thread since it seems to have started
happening when removing and adding a lot of OSDs. In fact I have removed
and added several times all the OSDs in the cluster, and mons have been
restarted several times but the load persists.
At debug_mon 20/5, I see endless lines of this which seems to be to do with
the osdmap:
2019-12-17 11:59:47.916098 7f27dfba1700 10 mon.mon1@1(peon) e4
handle_get_version mon_get_version(what=osdmap handle=2874836684) v1
2019-12-17 11:59:47.916139 7f27dfba1700 20 mon.mon1@1(peon) e4 _ms_dispatch
existing session 0x55ab61fb6300 for client.27824428 10.0.0.2:0/461841538
2019-12-17 11:59:47.916146 7f27dfba1700 20 mon.mon1@1(peon) e4 caps allow *
2019-12-17 11:59:47.916149 7f27dfba1700 20 is_capable service=mon command=
read on cap allow *
2019-12-17 11:59:47.916151 7f27dfba1700 20 allow so far , doing grant
allow *
2019-12-17 11:59:47.916152 7f27dfba1700 20 allow all
2019-12-17 11:59:47.916153 7f27dfba1700 10 mon.mon1@1(peon) e4
handle_get_version mon_get_version(what=osdmap handle=2871621985) v1
2019-12-17 11:59:47.916203 7f27dfba1700 20 mon.mon1@1(peon) e4 _ms_dispatch
existing session 0x55ab61d7c780 for client.27824430 10.0.0.2:0/898487246
2019-12-17 11:59:47.916210 7f27dfba1700 20 mon.mon1@1(peon) e4 caps allow *
2019-12-17 11:59:47.916213 7f27dfba1700 20 is_capable service=mon command=
read on cap allow *
2019-12-17 11:59:47.916215 7f27dfba1700 20 allow so far , doing grant
allow *
2019-12-17 11:59:47.916216 7f27dfba1700 20 allow all
2019-12-17 11:59:47.916217 7f27dfba1700 10 mon.mon1@1(peon) e4
handle_get_version mon_get_version(what=osdmap handle=2882637609) v1
2019-12-17 11:59:47.916254 7f27dfba1700 20 mon.mon1@1(peon) e4 _ms_dispatch
existing session 0x55ab62649c80 for client.27824431 10.0.0.2:0/972633098
2019-12-17 11:59:47.916262 7f27dfba1700 20 mon.mon1@1(peon) e4 caps allow *
2019-12-17 11:59:47.916266 7f27dfba1700 20 is_capable service=mon command=
read on cap allow *
2019-12-17 11:59:47.916268 7f27dfba1700 20 allow so far , doing grant
allow *
2019-12-17 11:59:47.916269 7f27dfba1700 20 allow all
Continuing to investigate.
Raf
On Tue, 17 Dec 2019 at 11:53, Sasha Litvak <alexander.v.litvak(a)gmail.com>
wrote:
Bryan, thank you. We are about to start testing
14.2.2 -> 14.2.5 upgrade,
so folks here are a bit cautious :-) We don't need to convert but may have
to rebuild few disks after an upgrade.
On Mon, Dec 16, 2019 at 3:57 PM Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
--
*Rafael Lopez*
Research Devops Engineer
Monash University eResearch Centre
E: rafael.lopez(a)monash.edu
Sasha,
I was able to get past it by restarting the ceph-mon processes every time
it got stuck, but that's not a very good solution for a production cluster.
Right now I'm trying to narrow down what is causing the problem.
Rebuilding the OSDs with BlueStore doesn't seem to be enough. I believe it
could be related to us using the extra space on the journal device as an
SSD-based OSD. During the conversion process I'm removing this SSD-based
OSD (since with BlueStore the omap data is ending up on the SSD anyways),
and I'm suspecting it might be causing this problem.
Bryan
On Dec 14, 2019, at 10:27 AM, Sasha Litvak <alexander.v.litvak(a)gmail.com>
wrote:
Notice: This email is from an external sender.
Bryan,
Were you able to resolve this? If yes, can you please share with the
list?
On Fri, Dec 13, 2019 at 10:08 AM Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
ceph-users mailing list --
ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Adding the dev list since it seems like a bug in
14.2.5.
I was able to capture the output from perf top:
21.58% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::list::append
20.90% libstdc++.so.6.0.19 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >
13.25% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::list::append
10.11% libstdc++.so.6.0.19 [.]
std::istream::sentry::sentry
8.94% libstdc++.so.6.0.19 [.] std::basic_ios<char,
std::char_traits<char> >::clear
3.24% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::ptr::unused_tail_length
1.69% libceph-common.so.0 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >@plt
1.63% libstdc++.so.6.0.19 [.]
std::istream::sentry::sentry@plt
1.21% [kernel] [k] __do_softirq
0.77% libpython2.7.so.1.0 [.] PyEval_EvalFrameEx
0.55% [kernel] [k]
_raw_spin_unlock_irqrestore
I increased mon debugging to 20 and nothing stuck out to me.
Bryan
_______________________________________________ On Dec 12, 2019, at 4:46 PM, Bryan Stillwell
<bstillwell(a)godaddy.com>
wrote:
On our test cluster after upgrading to 14.2.5 I'm having problems with
the
mons pegging a CPU core while moving data around. I'm currently
converting the OSDs from FileStore to BlueStore by marking the OSDs out in
multiple nodes, destroying the OSDs, and then recreating them with
ceph-volume lvm batch. This seems too get the ceph-mon process into a
state where it pegs a CPU core on one of the mons:
1764450 ceph 20 0 4802412 2.1g 16980 S 100.0 28.1 4:54.72
ceph-mon
Has anyone else run into this with 14.2.5 yet? I didn't see this
problem
while the cluster was running 14.2.4.
Thanks,
Bryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:59 a.m.
Mine was easier than I thought, turns out it was a bunch of rados client
connections stuck trying to do bench cleanup on a no-longer existing pool,
probably endlessly trying to find where the (also no longer existing) osds
it needs to talk to are.
On Tue, 17 Dec 2019 at 12:41, Rafael Lopez <rafael.lopez(a)monash.edu> wrote:
Hi guys,
I am running red hat ceph (basically luminous - ceph version
12.2.12-48.el7cp (26388d73d88602005946d4381cc5796d42904858)) and am seeing
something similar on our test cluster.
One of the mons is running at around 300% cpu non stop. It doesn't seem to
be the lead mon or one in particular, but the cpu load shifts to another
mon if high load mon is restarted.
I thought it might be related to this thread since it seems to have
started happening when removing and adding a lot of OSDs. In fact I have
removed and added several times all the OSDs in the cluster, and mons have
been restarted several times but the load persists.
At debug_mon 20/5, I see endless lines of this which seems to be to do
with the osdmap:
2019-12-17 11:59:47.916098 7f27dfba1700 10 mon.mon1@1(peon) e4
handle_get_version mon_get_version(what=osdmap handle=2874836684) v1
2019-12-17 11:59:47.916139 7f27dfba1700 20 mon.mon1@1(peon) e4
_ms_dispatch existing session 0x55ab61fb6300 for client.27824428
10.0.0.2:0/461841538
2019-12-17 11:59:47.916146 7f27dfba1700 20 mon.mon1@1(peon) e4 caps
allow *
2019-12-17 11:59:47.916149 7f27dfba1700 20 is_capable service=mon command=
read on cap allow *
2019-12-17 11:59:47.916151 7f27dfba1700 20 allow so far , doing grant
allow *
2019-12-17 11:59:47.916152 7f27dfba1700 20 allow all
2019-12-17 11:59:47.916153 7f27dfba1700 10 mon.mon1@1(peon) e4
handle_get_version mon_get_version(what=osdmap handle=2871621985) v1
2019-12-17 11:59:47.916203 7f27dfba1700 20 mon.mon1@1(peon) e4
_ms_dispatch existing session 0x55ab61d7c780 for client.27824430
10.0.0.2:0/898487246
2019-12-17 11:59:47.916210 7f27dfba1700 20 mon.mon1@1(peon) e4 caps
allow *
2019-12-17 11:59:47.916213 7f27dfba1700 20 is_capable service=mon command=
read on cap allow *
2019-12-17 11:59:47.916215 7f27dfba1700 20 allow so far , doing grant
allow *
2019-12-17 11:59:47.916216 7f27dfba1700 20 allow all
2019-12-17 11:59:47.916217 7f27dfba1700 10 mon.mon1@1(peon) e4
handle_get_version mon_get_version(what=osdmap handle=2882637609) v1
2019-12-17 11:59:47.916254 7f27dfba1700 20 mon.mon1@1(peon) e4
_ms_dispatch existing session 0x55ab62649c80 for client.27824431
10.0.0.2:0/972633098
2019-12-17 11:59:47.916262 7f27dfba1700 20 mon.mon1@1(peon) e4 caps
allow *
2019-12-17 11:59:47.916266 7f27dfba1700 20 is_capable service=mon command=
read on cap allow *
2019-12-17 11:59:47.916268 7f27dfba1700 20 allow so far , doing grant
allow *
2019-12-17 11:59:47.916269 7f27dfba1700 20 allow all
Continuing to investigate.
Raf
On Tue, 17 Dec 2019 at 11:53, Sasha Litvak <alexander.v.litvak(a)gmail.com>
wrote:
--
*Rafael Lopez*
Research Devops Engineer
Monash University eResearch Centre
T: +61 3 9905 9118
M: +61 (0)427682670 <%2B61%204%2027682%20670>
E: rafael.lopez(a)monash.edu
Bryan, thank you. We are about to start testing
14.2.2 -> 14.2.5
upgrade, so folks here are a bit cautious :-) We don't need to convert but
may have to rebuild few disks after an upgrade.
On Mon, Dec 16, 2019 at 3:57 PM Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
--
*Rafael Lopez*
Research Devops Engineer
Monash University eResearch Centre
E: rafael.lopez(a)monash.edu
Sasha,
I was able to get past it by restarting the ceph-mon processes every
time it got stuck, but that's not a very good solution for a production
cluster.
Right now I'm trying to narrow down what is causing the problem.
Rebuilding the OSDs with BlueStore doesn't seem to be enough. I believe it
could be related to us using the extra space on the journal device as an
SSD-based OSD. During the conversion process I'm removing this SSD-based
OSD (since with BlueStore the omap data is ending up on the SSD anyways),
and I'm suspecting it might be causing this problem.
Bryan
On Dec 14, 2019, at 10:27 AM, Sasha Litvak <alexander.v.litvak(a)gmail.com>
wrote:
Notice: This email is from an external sender.
Bryan,
Were you able to resolve this? If yes, can you please share with the
list?
On Fri, Dec 13, 2019 at 10:08 AM Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
ceph-users mailing list --
ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Adding the dev list since it seems like a bug in
14.2.5.
I was able to capture the output from perf top:
21.58% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::list::append
20.90% libstdc++.so.6.0.19 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >
13.25% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::list::append
10.11% libstdc++.so.6.0.19 [.]
std::istream::sentry::sentry
8.94% libstdc++.so.6.0.19 [.] std::basic_ios<char,
std::char_traits<char> >::clear
3.24% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::ptr::unused_tail_length
1.69% libceph-common.so.0 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >@plt
1.63% libstdc++.so.6.0.19 [.]
std::istream::sentry::sentry@plt
1.21% [kernel] [k] __do_softirq
0.77% libpython2.7.so.1.0 [.] PyEval_EvalFrameEx
0.55% [kernel] [k]
_raw_spin_unlock_irqrestore
I increased mon debugging to 20 and nothing stuck out to me.
Bryan
> On Dec 12, 2019, at 4:46 PM, Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
>
> On our test cluster after upgrading to 14.2.5 I'm having problems
with the mons pegging a CPU core while moving data around. I'm currently
converting the OSDs from FileStore to BlueStore by marking the OSDs out in
multiple nodes, destroying the OSDs, and then recreating them with
ceph-volume lvm batch. This seems too get the ceph-mon process into a
state where it pegs a CPU core on one of the mons:
>
> 1764450 ceph 20 0 4802412 2.1g 16980 S 100.0 28.1 4:54.72
ceph-mon
>
> Has anyone else run into this with 14.2.5 yet? I didn't see this
problem while the cluster was running 14.2.4.
>
> Thanks,
> Bryan
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________ 
24 Feb
24 Feb
3:51 p.m.
Hi Bryan,
Did you ever learn more about this, or see it again?
I'm facing 100% ceph-mon CPU usage now, and putting my observations
here: https://tracker.ceph.com/issues/42830
Cheers, Dan
On Mon, Dec 16, 2019 at 10:58 PM Bryan Stillwell <bstillwell(a)godaddy.com> wrote:
>
> Sasha,
>
> I was able to get past it by restarting the ceph-mon processes every time it got
stuck, but that's not a very good solution for a production cluster.
>
> Right now I'm trying to narrow down what is causing the problem. Rebuilding the
OSDs with BlueStore doesn't seem to be enough. I believe it could be related to us
using the extra space on the journal device as an SSD-based OSD. During the conversion
process I'm removing this SSD-based OSD (since with BlueStore the omap data is ending
up on the SSD anyways), and I'm suspecting it might be causing this problem.
>
> Bryan
>
> On Dec 14, 2019, at 10:27 AM, Sasha Litvak <alexander.v.litvak(a)gmail.com>
wrote:
>
> Notice: This email is from an external sender.
>
> Bryan,
>
> Were you able to resolve this? If yes, can you please share with the list?
>
> On Fri, Dec 13, 2019 at 10:08 AM Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
>>
>> Adding the dev list since it seems like a bug in 14.2.5.
>>
>> I was able to capture the output from perf top:
>>
>> 21.58% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::list::append
>> 20.90% libstdc++.so.6.0.19 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >
>> 13.25% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::list::append
>> 10.11% libstdc++.so.6.0.19 [.] std::istream::sentry::sentry
>> 8.94% libstdc++.so.6.0.19 [.] std::basic_ios<char,
std::char_traits<char> >::clear
>> 3.24% libceph-common.so.0 [.]
ceph::buffer::v14_2_0::ptr::unused_tail_length
>> 1.69% libceph-common.so.0 [.] std::getline<char,
std::char_traits<char>, std::allocator<char> >@plt
>> 1.63% libstdc++.so.6.0.19 [.] std::istream::sentry::sentry@plt
>> 1.21% [kernel] [k] __do_softirq
>> 0.77% libpython2.7.so.1.0 [.] PyEval_EvalFrameEx
>> 0.55% [kernel] [k] _raw_spin_unlock_irqrestore
>>
>> I increased mon debugging to 20 and nothing stuck out to me.
>>
>> Bryan
>>
>> > On Dec 12, 2019, at 4:46 PM, Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
>> >
>> > On our test cluster after upgrading to 14.2.5 I'm having problems with
the mons pegging a CPU core while moving data around. I'm currently converting the
OSDs from FileStore to BlueStore by marking the OSDs out in multiple nodes, destroying the
OSDs, and then recreating them with ceph-volume lvm batch. This seems too get the
ceph-mon process into a state where it pegs a CPU core on one of the mons:
>> >
>> > 1764450 ceph 20 0 4802412 2.1g 16980 S 100.0 28.1 4:54.72
ceph-mon
>> >
>> > Has anyone else run into this with 14.2.5 yet? I didn't see this
problem while the cluster was running 14.2.4.
>> >
>> > Thanks,
>> > Bryan
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
16 Dec
16 Dec
4:25 p.m.
On Thu, Dec 12, 2019 at 11:46:19PM +0000, Bryan Stillwell <bstillwell(a)godaddy.com>
wrote:
On our test cluster after upgrading to 14.2.5 I'm
having problems with the mons pegging a CPU core while moving data around. I'm
currently converting the OSDs from FileStore to BlueStore by marking the OSDs out in
multiple nodes, destroying the OSDs, and then recreating them with ceph-volume lvm batch.
This seems too get the ceph-mon process into a state where it pegs a CPU core on one of
the mons:
I had a similar issue in a partially virtualized test cluster with
14.2.4 when I removed many OSDs. I resovled it either by adding more RAM
to the MON VMs or by restarting the MONs multiple times. The VM changed
from 2gb -> 4gb. ceph-mon in this small test cluster normally uses
around 100-200mb; during the issue it used (at least) 1.7gb for a short
time. I can't tell you exactly which of the two resolved it since adding
more RAM to the VM required a reboot and thus a restart of the affected
process.
Also my cluster was unable to elect a MGR and it was pretty much dead
until the MONs recovered again. I'm not sure if that is the case with
your cluster as well.
HTH,
Florian
1522
days inactive
1596
days old
8 comments
5 participants
participants (5)
-
Bryan Stillwell -
Dan van der Ster -
Florian Pritz -
Rafael Lopez -
Sasha Litvak