2 Jul
2021
2 Jul
'21
2:24 p.m.
Hi,
Want to have logs from cluster on Graylog but seems like CEPH send empty
"host" field. Any one can help ?
CEPH 16.2.3
# ceph config dump | grep graylog
global advanced clog_to_graylog true
global advanced clog_to_graylog_host xx.xx.xx.xx
global basic err_to_graylog true
global basic log_graylog_host xx.xx.xx.xx *
global basic log_to_graylog true
I see that my Graylog is hit by traffic from ceph on port 12201 udp and
parsed by GELF udp
Grylog logs:
2021-07-01 12:16:57,355 ERROR:
org.graylog2.shared.buffers.processors.DecodingProcessor - Error
processing message RawMessage{id=3ad5c6a1-da66-11eb-a55c-0242ac120005,
messageQueueId=2810784, codec=gelf, payloadSize=340,
timestamp=2021-07-01T12:16:57.354Z, remoteAddress=/xx.xx.xx.xx:34049}
java.lang.IllegalArgumentException: GELF message
<3ad5c6a1-da66-11eb-a55c-0242ac120005> (received from
<xx.xx.xx.xx:34049>) has empty mandatory "host" field.
at
org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:247)
~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140)
~[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153)
~[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94)
[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90)
[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47)
[graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143)
[graylog.jar:?]
at
com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66)
[graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Best regards Milosz
5 Jul
5 Jul
8:11 a.m.
Hi Milosz,
I don't have any experience with the settings you're using so can't
help there, but I do log to graylog via syslog.
This is what I do, in case it's helpful as a workaround.
In ceph.conf global section or config db:
log to syslog = true
err to syslog = true
in rsyslog.conf add preserve hostname to get fqdn hostnames
$PreserveFQDN on
in rsyslog.d create a file to catch all. You could, of course, just
specify ceph related logs here if you don't want host logs.
*.* @ip_address_of_graylog
Regards,
Rich
On Fri, 2 Jul 2021 at 17:25, <milosz(a)dev.datacrunch.io> wrote:
>
> Hi,
>
> Want to have logs from cluster on Graylog but seems like CEPH send empty
> "host" field. Any one can help ?
>
> CEPH 16.2.3
> # ceph config dump | grep graylog
> global advanced clog_to_graylog true
> global advanced clog_to_graylog_host xx.xx.xx.xx
> global basic err_to_graylog true
> global basic log_graylog_host xx.xx.xx.xx *
> global basic log_to_graylog true
>
> I see that my Graylog is hit by traffic from ceph on port 12201 udp and
> parsed by GELF udp
>
> Grylog logs:
>
> 2021-07-01 12:16:57,355 ERROR:
> org.graylog2.shared.buffers.processors.DecodingProcessor - Error
> processing message RawMessage{id=3ad5c6a1-da66-11eb-a55c-0242ac120005,
> messageQueueId=2810784, codec=gelf, payloadSize=340,
> timestamp=2021-07-01T12:16:57.354Z, remoteAddress=/xx.xx.xx.xx:34049}
> java.lang.IllegalArgumentException: GELF message
> <3ad5c6a1-da66-11eb-a55c-0242ac120005> (received from
> <xx.xx.xx.xx:34049>) has empty mandatory "host" field.
> at
> org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:247)
> ~[graylog.jar:?]
> at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140)
> ~[graylog.jar:?]
> at
>
org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153)
> ~[graylog.jar:?]
> at
>
org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94)
> [graylog.jar:?]
> at
>
org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90)
> [graylog.jar:?]
> at
>
org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47)
> [graylog.jar:?]
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143)
> [graylog.jar:?]
> at
>
com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66)
> [graylog.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
>
> Best regards Milosz
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
8 Jul
8 Jul
1:59 a.m.
Hi Milosz,
The host field should be set to the value of the 'host' global
config attribute in the Graylog backend [0]. Usually set right
after init or on config changes. Not having that field in the GELF
messages suggests that something is not right.
With a vstart cluster and "fake graylog" [1] I can confirm: Some -
not all - messages have empty 'host' and sometimes '_fsid' fields.
They seem to come from subsystems mon, ms and paxos. (My sample
size is quite small though; might be more).
Can you confirm that at least some messages have a host field and
end up in your Graylog?
Best,
marcel
[0] https://github.com/ceph/ceph/blob/master/src/common/Graylog.cc
[1] socat -u udp4-recvfrom:12201,reuseaddr,fork system:"pigz -d |
jq"
On 2021-07-05 11:11:50 Richard Bade <hitrich(a)gmail.com> wrote:
Hi Milosz,
I don't have any experience with the settings you're using so
can't
help there, but I do log to graylog via syslog.
This is what I do, in case it's helpful as a workaround.
In ceph.conf global section or config db:
log to syslog = true
err to syslog = true
in rsyslog.conf add preserve hostname to get fqdn hostnames
$PreserveFQDN on
in rsyslog.d create a file to catch all. You could, of course,
just
specify ceph related logs here if you don't want host logs.
*.* @ip_address_of_graylog
Regards,
Rich
On Fri, 2 Jul 2021 at 17:25, <milosz(a)dev.datacrunch.io> wrote:
Hi,
Want to have logs from cluster on Graylog but seems like CEPH
send empty
"host" field. Any one can help ?
CEPH 16.2.3
# ceph config dump | grep graylog
global advanced clog_to_graylog true
global advanced clog_to_graylog_host xx.xx.xx.xx
global basic err_to_graylog true
global basic log_graylog_host xx.xx.xx.xx *
global basic log_to_graylog true
I see that my Graylog is hit by traffic from ceph on port 12201
udp and
parsed by GELF udp
Grylog logs:
2021-07-01 12:16:57,355 ERROR:
org.graylog2.shared.buffers.processors.DecodingProcessor -
Error
processing message
RawMessage{id=3ad5c6a1-da66-11eb-a55c-0242ac120005,
messageQueueId=2810784, codec=gelf, payloadSize=340,
timestamp=2021-07-01T12:16:57.354Z,
remoteAddress=/xx.xx.xx.xx:34049}
java.lang.IllegalArgumentException: GELF message
<3ad5c6a1-da66-11eb-a55c-0242ac120005> (received from
<xx.xx.xx.xx:34049>) has empty mandatory "host" field.
at
org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:247)
~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140)
~[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153)
~[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94)
[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90)
[graylog.jar:?]
at
org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47)
[graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143)
[graylog.jar:?]
at
com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66)
[graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Best regards Milosz
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1027
days inactive
1032
days old
2 comments
3 participants
participants (3)
-
Marcel Lauhoff -
milosz＠dev.datacrunch.io -
Richard Bade