Hi Milosz, The host field should be set to the value of the 'host' global config attribute in the Graylog backend [0]. Usually set right after init or on config changes. Not having that field in the GELF messages suggests that something is not right. With a vstart cluster and "fake graylog" [1] I can confirm: Some - not all - messages have empty 'host' and sometimes '_fsid' fields. They seem to come from subsystems mon, ms and paxos. (My sample size is quite small though; might be more). Can you confirm that at least some messages have a host field and end up in your Graylog? Best, marcel [0] https://github.com/ceph/ceph/blob/master/src/common/Graylog.cc [1] socat -u udp4-recvfrom:12201,reuseaddr,fork system:"pigz -d | jq" On 2021-07-05 11:11:50 Richard Bade <hitrich@gmail.com> wrote:

Hi Milosz, I don't have any experience with the settings you're using so can't help there, but I do log to graylog via syslog. This is what I do, in case it's helpful as a workaround. In ceph.conf global section or config db: log to syslog = true err to syslog = true

in rsyslog.conf add preserve hostname to get fqdn hostnames $PreserveFQDN on

in rsyslog.d create a file to catch all. You could, of course, just specify ceph related logs here if you don't want host logs. *.* @ip_address_of_graylog

Regards, Rich

On Fri, 2 Jul 2021 at 17:25, <milosz@dev.datacrunch.io> wrote:

...

Hi,

Want to have logs from cluster on Graylog but seems like CEPH send empty "host" field. Any one can help ?

CEPH 16.2.3 # ceph config dump | grep graylog global advanced clog_to_graylog true global advanced clog_to_graylog_host xx.xx.xx.xx global basic err_to_graylog true global basic log_graylog_host xx.xx.xx.xx * global basic log_to_graylog true

I see that my Graylog is hit by traffic from ceph on port 12201 udp and parsed by GELF udp

Grylog logs:

2021-07-01 12:16:57,355 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=3ad5c6a1-da66-11eb-a55c-0242ac120005, messageQueueId=2810784, codec=gelf, payloadSize=340, timestamp=2021-07-01T12:16:57.354Z, remoteAddress=/xx.xx.xx.xx:34049} java.lang.IllegalArgumentException: GELF message <3ad5c6a1-da66-11eb-a55c-0242ac120005> (received from <xx.xx.xx.xx:34049>) has empty mandatory "host" field. at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:247) ~[graylog.jar:?] at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) ~[graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?] at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]

Best regards Milosz _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io