Maybe the debugging steps in that insights tracker can be helpful
anyway:
https://tracker.ceph.com/issues/39955
-- dan
On Thu, Feb 25, 2021 at 9:27 PM Janek Bevendorff
<janek.bevendorff(a)uni-weimar.de> wrote:
>
> Thanks for the tip, but I do not have degraded PGs and the module is already
disabled.
>
>
> On 25. Feb 2021, at 21:17, Seena Fallah <seenafallah(a)gmail.com> wrote:
>
> I had the same problem in my cluster and it was because of insights mgr module that
was storing lots of data to the RocksDB because mu cluster was degraded.
> If you have degraded pgs try to disable insights module.
>
> On Thu, Feb 25, 2021 at 11:40 PM Dan van der Ster <dan(a)vanderster.com> wrote:
>>
>> > "source": "osd.104...
>>
>> What's happening on that osd? Is it something new which corresponds to when
>> your mon started growing? Are other OSDs also flooding the mons with logs?
>>
>> I'm mobile so can't check... Are those logging configs the defaults? If
not
>> .... revert to default...
>>
>> BTW do your mons have stable quorum or are they flapping with this load?
>>
>> .. dan
>>
>>
>>
>> On Thu, Feb 25, 2021, 8:58 PM Janek Bevendorff <
>> janek.bevendorff(a)uni-weimar.de> wrote:
>>
>> > Thanks, Dan.
>> >
>> > On the first MON, the command doesn’t even return, but I was able to get a
>> > dump from the one I restarted most recently. The oldest ops look like this:
>> >
>> > {
>> > "description": "log(1000 entries from seq
17876238 at
>> > 2021-02-25T15:13:20.306487+0100)",
>> > "initiated_at":
"2021-02-25T20:40:34.698932+0100",
>> > "age": 183.762551121,
>> > "duration": 183.762599201,
>> > "type_data": {
>> > "events": [
>> > {
>> > "time":
"2021-02-25T20:40:34.698932+0100",
>> > "event": "initiated"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.698636+0100",
>> > "event": "throttled"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.698932+0100",
>> > "event": "header_read"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.701407+0100",
>> > "event": "all_read"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.701455+0100",
>> > "event": "dispatched"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.701458+0100",
>> > "event": "mon:_ms_dispatch"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.701459+0100",
>> > "event": "mon:dispatch_op"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.701459+0100",
>> > "event": "psvc:dispatch"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.701490+0100",
>> > "event":
"logm:wait_for_readable"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.701491+0100",
>> > "event":
"logm:wait_for_readable/paxos"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.701496+0100",
>> > "event":
"paxos:wait_for_readable"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.989198+0100",
>> > "event": "callback finished"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.989199+0100",
>> > "event": "psvc:dispatch"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.989208+0100",
>> > "event":
"logm:preprocess_query"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.989208+0100",
>> > "event": "logm:preprocess_log"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.989278+0100",
>> > "event":
"forward_request_leader"
>> > },
>> > {
>> > "time":
"2021-02-25T20:40:34.989344+0100",
>> > "event": "forwarded"
>> > },
>> > {
>> > "time":
"2021-02-25T20:41:58.658022+0100",
>> > "event": "resend forwarded message to
leader"
>> > },
>> > {
>> > "time":
"2021-02-25T20:42:27.735449+0100",
>> > "event": "resend forwarded message to
leader"
>> > }
>> > ],
>> > "info": {
>> > "seq": 41550,
>> > "src_is_mon": false,
>> > "source": "osd.104
v2:XXX:6864/16579",
>> > "forwarded_to_leader": true
>> > }
>> >
>> >
>> > Any idea what that might be about? Almost looks like this:
>> >
https://tracker.ceph.com/issues/24180
>> > I set debug_mon to 0, but I keep getting a lot of log spill in journals.
>> > It’s about 1-2 messages per second, mostly RocksDB stuff, but nothing that
>> > actually looks serious or even log-worthy. I noticed that before that
>> > despite logging being set to warning level, the cluster log keeps being
>> > written to the MON log. But it shouldn’t cause such massive stability
>> > issues, should it? The date on the log op is also weird. 15:13+0100 was
>> > hours ago.
>> >
>> > Here’s my log config:
>> >
>> > global advanced clog_to_syslog_level
>> > warning
>> > global basic err_to_syslog
>> > true
>> > global basic log_to_file
>> > false
>> > global basic log_to_stderr
>> > false
>> > global basic log_to_syslog
>> > true
>> > global advanced mon_cluster_log_file_level
>> > error
>> > global advanced mon_cluster_log_to_file
>> > false
>> > global advanced mon_cluster_log_to_stderr
>> > false
>> > global advanced mon_cluster_log_to_syslog
>> > false
>> > global advanced
>> > mon_cluster_log_to_syslog_level warning
>> >
>> >
>> >
>> > Ceph version is 15.2.8.
>> >
>> > Janek
>> >
>> >
>> > On 25. Feb 2021, at 20:33, Dan van der Ster <dan(a)vanderster.com>
wrote:
>> >
>> > ceph daemon mon.`hostname -s` ops
>> >
>> > That should show you the accumulating ops.
>> >
>> > .. dan
>> >
>> >
>> > On Thu, Feb 25, 2021, 8:23 PM Janek Bevendorff <
>> > janek.bevendorff(a)uni-weimar.de> wrote:
>> >
>> >> Hi,
>> >>
>> >> All of a sudden, we are experiencing very concerning MON behaviour. We
>> >> have five MONs and all of them have thousands up to tens of thousands
of
>> >> slow ops, the oldest one blocking basically indefinitely (at least the
>> >> timer keeps creeping up). Additionally, the MON stores keep inflating
>> >> heavily. Under normal circumstances we have about 450-550MB there.
Right
>> >> now its 27GB and growing (rapidly).
>> >>
>> >> I tried restarting all MONs, I disabled auto-scaling (just in case) and
>> >> checked the system load and hardware. I also restarted the MGR and MDS
>> >> daemons, but to no avail.
>> >>
>> >> Is there any way I can debug this properly? I can’t seem to find how I
>> >> can actually view what ops are causing this and what client (if any) may
be
>> >> responsible for it.
>> >>
>> >> Thanks
>> >> Janek
>> >> _______________________________________________
>> >> ceph-users mailing list -- ceph-users(a)ceph.io
>> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>> >>
>> >
>> >
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>