Ceph OSD Jaeger tracing: errors and no end-to-end traces
Hi Ceph community, We enabled the native Jaeger tracing on the OSDs of our Ceph 19.2.3 cluster to get distributed tracing. Issues: 1. Every OSD continuously logs this following error, [Error] File: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.3/rpm/el9/BUILD/ceph-19.2.3/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/src/recordable.cc:95[TRACE JAEGER Exporter] SetAttribute() failed, attribute type not supported 2. In Jaeger we only see two standalone spans per op (op requested/created and enqueue). They are not connected to each other or to any client-side trace. Questions 1. Is the SetAttribute() error a known issue, and is there a fix or workaround? 2. What is the intended use of these two spans? Is there a way to get full end-to-end traces in current releases? If anyone has faced this or knows a fix, please let us know. Thanks & Regards, Kathirvel T Associate R&D Engineer 9025610912 mailto:manoj.kumar@zybisys.com kathirvel.t@zybisys.com https://www.zybisys.com https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."
On Thu, Oct 1, 2026 at 7:28 PM Kathirvel T <ceph-users@ceph.io> wrote:
Hi Ceph community,
We enabled the native Jaeger tracing on the OSDs of our Ceph 19.2.3 cluster to get distributed tracing.
Issues:
1. Every OSD continuously logs this following error,
[Error] File: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.3/rpm/el9/BUILD/ceph-19.2.3/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/src/recordable.cc:95[TRACE JAEGER Exporter] SetAttribute() failed, attribute type not supported
2. In Jaeger we only see two standalone spans per op (op requested/created and enqueue). They are not connected to each other or to any client-side trace.
Questions
1. Is the SetAttribute() error a known issue, and is there a fix or workaround?
The error is a bug. enqueue_op adds the owner attribute as a uint64_t,
which the Jaeger Thrift exporter doesn't support, so every queued op logs that line. It's harmless (only that one tag is dropped, the span is still exported), but there's no option to silence it short of disabling tracing. The fix is a one-line cast to int64_t; I'll send a PR and request a Squid backport. Until then, filtering that message out in journald or your log shipper is the practical workaround.
2. What is the intended use of these two spans? Is there a way to get full end-to-end traces in current releases?
Those two spans are all the OSD emits today, to keep the resource overhead
low : op-request-created when the op arrives and enqueue_op when it's queued to the PG. They do belong to the same trace, but it might not be timing nicely enough to be stitched together. Only RGW propagates a trace context to the OSD (tracing has to be enabled on the RGW too); RBD and CephFS clients don't right now, so their ops always start a new trace. so end-to-end traces aren't available in Squid for these 2 clients.
That gap is what I'm working on: the OSD builds a trace for each op slower than a threshold, from the events the op tracker already records, with the primary's op and the replicas' sub-ops in one trace, so you see which replica held up a write. Ops under the threshold cost nothing extra, and spans can go over OTLP to Jaeger v2. I'll share the draft https://github.com/ceph/ceph/pull/72328
In the meantime, the same timeline is available with historic ops : ceph daemon osd.N dump_historic_ops , If you're tracing for latency rather than to see the call path, I'd turn jaeger_tracing_enable back off, since it adds CPU per op and these two spans might not give full picture as of now.
hope this helps!
If anyone has faced this or knows a fix, please let us know.
Thanks & Regards,
Kathirvel T
Associate R&D Engineer
9025610912 <(902)%20561-0912> mailto:manoj.kumar@zybisys.com
kathirvel.t@zybisys.com
https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender.
E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Deepika, Thank you for the detailed explanation. This provides us much better understanding of how tracing in ceph currently works and the context behind the SetAttribute() error. We look forward to seeing your draft integrated for full end-to-end trace visibility. Thanks & Regards, Kathirvel T Associate R&D Engineer 9025610912 mailto:manoj.kumar@zybisys.com kathirvel.t@zybisys.com https://www.zybisys.com https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ From: Deepika Upadhyay <ceph-users@ceph.io> To: "Kathirvel T"<kathirvel.t@zybisys.com> Cc: "ceph-users"<ceph-users@ceph.io>, "dev"<dev@ceph.io>, "Vasanth G"<vasanth.g@zybisys.com> Date: Fri, 02 Oct 2026 12:14:56 +0530 Subject: [ceph-users] Re: Ceph OSD Jaeger tracing: errors and no end-to-end traces On Thu, Oct 1, 2026 at 7:28 PM Kathirvel T < mailto:ceph-users@ceph.io > wrote:
Hi Ceph community,
We enabled the native Jaeger tracing on the OSDs of our Ceph 19.2.3 cluster to get distributed tracing.
Issues:
1. Every OSD continuously logs this following error,
[Error] File: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.2.3/rpm/el9/BUILD/ceph-19.2.3/src/jaegertracing/opentelemetry-cpp/exporters/jaeger/src/recordable.cc:95[TRACE JAEGER Exporter] SetAttribute() failed, attribute type not supported
2. In Jaeger we only see two standalone spans per op (op requested/created and enqueue). They are not connected to each other or to any client-side trace.
Questions
1. Is the SetAttribute() error a known issue, and is there a fix or workaround?
The error is a bug. enqueue_op adds the owner attribute as a uint64_t,
which the Jaeger Thrift exporter doesn't support, so every queued op logs that line. It's harmless (only that one tag is dropped, the span is still exported), but there's no option to silence it short of disabling tracing. The fix is a one-line cast to int64_t; I'll send a PR and request a Squid backport. Until then, filtering that message out in journald or your log shipper is the practical workaround.
2. What is the intended use of these two spans? Is there a way to get full end-to-end traces in current releases?
Those two spans are all the OSD emits today, to keep the resource overhead
low : op-request-created when the op arrives and enqueue_op when it's queued to the PG. They do belong to the same trace, but it might not be timing nicely enough to be stitched together. Only RGW propagates a trace context to the OSD (tracing has to be enabled on the RGW too); RBD and CephFS clients don't right now, so their ops always start a new trace. so end-to-end traces aren't available in Squid for these 2 clients.
That gap is what I'm working on: the OSD builds a trace for each op slower than a threshold, from the events the op tracker already records, with the primary's op and the replicas' sub-ops in one trace, so you see which replica held up a write. Ops under the threshold cost nothing extra, and spans can go over OTLP to Jaeger v2. I'll share the draft https://github.com/ceph/ceph/pull/72328
In the meantime, the same timeline is available with historic ops : ceph daemon osd.N dump_historic_ops , If you're tracing for latency rather than to see the call path, I'd turn jaeger_tracing_enable back off, since it adds CPU per op and these two spans might not give full picture as of now.
hope this helps!
If anyone has faced this or knows a fix, please let us know.
Thanks & Regards,
Kathirvel T
Associate R&D Engineer
9025610912 <(902)%20561-0912> mailto: mailto:manoj.kumar@zybisys.com
mailto:kathirvel.t@zybisys.com
https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender.
E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."
_______________________________________________ ceph-users mailing list -- mailto:ceph-users@ceph.io To unsubscribe send an email to mailto:ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- mailto:ceph-users@ceph.io To unsubscribe send an email to mailto:ceph-users-leave@ceph.io Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."
participants (2)
-
Deepika Upadhyay
-
Kathirvel T