Help with Jaeger Tracing for Ceph RGW in Kubernetes (Rook-Ceph Deployment)
Hi Ceph Community, I'm currently running a Ceph cluster inside a Kubernetes environment using Rook-Ceph, and I'm working on enabling distributed tracing for Ceph RGW. As per the https://docs.ceph.com/en/latest/jaegertracing/ , I used the following docker run command to deploy a Jaeger all-in-one instance for testing: docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ -e COLLECTOR_OTLP_ENABLED=true \ -p 6799:6799/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 14269:14269 \ -p 9411:9411 \ jaegertracing/all-in-one:latest \ --processor.jaeger-compact.server-host-port=6799 However, when using the latest Jaeger version, I get the following error: unknown flag: --processor.jaeger-compact.server-host-port After researching, I discovered that this flag is only available in older versions of Jaeger (e.g., v1.27) and is no longer supported in Jaeger v2+. I then tried using a specific older Jaeger version (jaegertracing/all-in-one:1.27) that supports the flag. The container runs fine and listens on port 6799 as expected, but I still don’t see any RGW traces appearing in the Jaeger UI. My Setup: Ceph cluster deployed via Rook in Kubernetes Ceph RGW is running as a pod inside the cluster Jaeger all-in-one is running externally (for testing) Jaeger port 6799/udp is exposed and forwarded Tracing is enabled in Ceph config Questions: Could the issue be related to the Ceph RGW pod not being able to send UDP traces to the Jaeger container? Is there a specific configuration needed inside the RGW pod or Rook-Ceph to make sure tracing data is being sent correctly? Should I consider using the default Jaeger port 6831/udp instead of 6799/udp, and configure Ceph accordingly? Are there any known compatibility issues between Ceph RGW tracing and Jaeger v1.x? Could you suggest some troubleshooting steps to help verify whether traces are actually being sent from RGW and received by Jaeger? Any advice or working examples from a similar Kubernetes-based setup would be greatly appreciated. Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."
hey Ashok, Have you tried enabling debug logs to check if the tracer is getting initialized and generating traces? You can do this by running: ceph config set global debug_trace 20 Then, try grepping the logs for relevant trace information. You can also refer to tracer.cc <https://github.com/ceph/ceph/blob/main/src/common/tracer.cc> to see where the debug points are. If you are using Rook-Ceph, you’ll need to inject a Jaeger sidecar into the OSD and RGW pods. Yuval and I worked on using Rook-Ceph with Jaeger for our Cephalocon presentation last year [1], and you can take inspiration from that approach if you’re using Rook-Ceph [2]. [1] https://www.youtube.com/watch?v=Yo9yB3KYBwo [2] https://gist.github.com/ideepika/42f87206cbde8a4bff78f93af72ee642 Feel free to reach out if you have any questions. Best, Deepika On Thu, Sep 25, 2025 at 11:41 AM Ashok Kumar M <ashok.m@zybisys.com> wrote:
Hi Ceph Community,
I'm currently running a Ceph cluster inside a Kubernetes environment using *Rook-Ceph*, and I'm working on enabling *distributed tracing for Ceph RGW*. As per the https://docs.ceph.com/en/latest/jaegertracing/ , I used the following docker run command to deploy a Jaeger all-in-one instance for testing:
docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ -e COLLECTOR_OTLP_ENABLED=true \ -p 6799:6799/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 14269:14269 \ -p 9411:9411 \ jaegertracing/all-in-one:latest \ --processor.jaeger-compact.server-host-port=6799
However, when using the *latest Jaeger version*, I get the following error: unknown flag: --processor.jaeger-compact.server-host-port
After researching, I discovered that this flag is only available in *older versions of Jaeger (e.g., v1.27)* and is no longer supported in Jaeger v2+.
I then tried using a specific older Jaeger version ( jaegertracing/all-in-one:1.27) that supports the flag. The container runs fine and listens on port 6799 as expected, but I *still don’t see any RGW traces appearing in the Jaeger UI*. My Setup:
-
Ceph cluster deployed via Rook in Kubernetes -
Ceph RGW is running as a pod inside the cluster -
Jaeger all-in-one is running externally (for testing) -
Jaeger port 6799/udp is exposed and forwarded -
Tracing is enabled in Ceph config
Questions:
1.
Could the issue be related to the Ceph RGW pod not being able to send UDP traces to the Jaeger container? 2.
Is there a specific configuration needed inside the RGW pod or Rook-Ceph to make sure tracing data is being sent correctly? 3.
Should I consider using the default Jaeger port 6831/udp instead of 6799/udp, and configure Ceph accordingly? 4.
Are there any known compatibility issues between Ceph RGW tracing and Jaeger v1.x? 5.
Could you suggest some *troubleshooting steps* to help verify whether traces are actually being sent from RGW and received by Jaeger?
Any advice or working examples from a similar Kubernetes-based setup would be greatly appreciated.
------------------------------ *Disclaimer :** The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender.*
*E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."*
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi Deepika, Thank you sincerely for your response and for offering to help with any further doubts. I will try the approach you suggested, and I’m confident it will help me complete this progress successfully. Best regards, Ashok M ---- On Fri, 26 Sep 2025 01:38:41 +0530 Deepika Upadhyay <deepikaupadhyay01@gmail.com> wrote --- hey Ashok, Have you tried enabling debug logs to check if the tracer is getting initialized and generating traces? You can do this by running: ceph config set global debug_trace 20 Then, try grepping the logs for relevant trace information. You can also refer to https://github.com/ceph/ceph/blob/main/src/common/tracer.cc to see where the debug points are. If you are using Rook-Ceph, you’ll need to inject a Jaeger sidecar into the OSD and RGW pods. Yuval and I worked on using Rook-Ceph with Jaeger for our Cephalocon presentation last year [1], and you can take inspiration from that approach if you’re using Rook-Ceph [2]. [1] https://www.youtube.com/watch?v=Yo9yB3KYBwo [2] https://gist.github.com/ideepika/42f87206cbde8a4bff78f93af72ee642 Feel free to reach out if you have any questions. Best, Deepika On Thu, Sep 25, 2025 at 11:41 AM Ashok Kumar M < mailto:ashok.m@zybisys.com > wrote: Hi Ceph Community, I'm currently running a Ceph cluster inside a Kubernetes environment using Rook-Ceph, and I'm working on enabling distributed tracing for Ceph RGW. As per the https://docs.ceph.com/en/latest/jaegertracing/ , I used the following docker run command to deploy a Jaeger all-in-one instance for testing: docker run -d --name jaeger \ -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \ -e COLLECTOR_OTLP_ENABLED=true \ -p 6799:6799/udp \ -p 6832:6832/udp \ -p 5778:5778 \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ -p 14250:14250 \ -p 14268:14268 \ -p 14269:14269 \ -p 9411:9411 \ jaegertracing/all-in-one:latest \ --processor.jaeger-compact.server-host-port=6799 However, when using the latest Jaeger version, I get the following error: unknown flag: --processor.jaeger-compact.server-host-port After researching, I discovered that this flag is only available in older versions of Jaeger (e.g., v1.27) and is no longer supported in Jaeger v2+. I then tried using a specific older Jaeger version (jaegertracing/all-in-one:1.27) that supports the flag. The container runs fine and listens on port 6799 as expected, but I still don’t see any RGW traces appearing in the Jaeger UI. My Setup: Ceph cluster deployed via Rook in Kubernetes Ceph RGW is running as a pod inside the cluster Jaeger all-in-one is running externally (for testing) Jaeger port 6799/udp is exposed and forwarded Tracing is enabled in Ceph config Questions: Could the issue be related to the Ceph RGW pod not being able to send UDP traces to the Jaeger container? Is there a specific configuration needed inside the RGW pod or Rook-Ceph to make sure tracing data is being sent correctly? Should I consider using the default Jaeger port 6831/udp instead of 6799/udp, and configure Ceph accordingly? Are there any known compatibility issues between Ceph RGW tracing and Jaeger v1.x? Could you suggest some troubleshooting steps to help verify whether traces are actually being sent from RGW and received by Jaeger? Any advice or working examples from a similar Kubernetes-based setup would be greatly appreciated. Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email." _______________________________________________ Dev mailing list -- mailto:dev@ceph.io To unsubscribe send an email to mailto:dev-leave@ceph.io Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."
participants (2)
-
Ashok Kumar M
-
Deepika Upadhyay