Confirmed for 19.2.0 in a lab cluster. Zitat von Eugen Block <eblock@nde.ag>:

I see the same as Sanjay, there's no movement in the graph "Recovery Throughput", this is on 19.1.1. If I switch to the classic dashboard though (ceph config set mgr mgr/dashboard/FEATURE_TOGGLE_DASHBOARD false), I see recovery traffic in the dashboard as well.

I'll upgrade to 19.2.0 and retry.

Zitat von Frédéric Nass <frederic.nass@univ-lorraine.fr>:

...

Hi Sanjay,

I've just checked the dashboard of a v19.2.0 cluster, and the recovery throughput is displayed correctly, as shown in the screenshot here [1]. You might want to consider redeploying the dashboard.

Regards, Frédéric.

[1] https://docs.ceph.com/en/latest/mgr/dashboard/

----- Le 19 Oct 24, à 19:23, Sanjay Mohan sanjaymohan@am.amrita.edu a écrit :

...

Dear Ceph Users, I hope this message finds you well. I recently performed an upgrade of my Ceph cluster, moving through the following versions:

* 17.2.7 >> 18.2.0 >> 18.2.2 >> 18.2.4 >> 19.2.0 (Squid)

After successfully upgrading to Ceph 19.2.0, I noticed an issue where the recovery throughput is no longer visible in the new Ceph dashboard. However, Old dashboard metrics and features seem to be working as expected. It is important to note that the recovery throughput was displayed properly in the previous version of the Ceph dashboard. I am using Cephadm for the installation, not Rook. Current behavior:

* Recovery throughput metrics are not displayed in the new dashboard after upgrading to 19.2.0.

Expected behavior:

* The recovery throughput should be visible, as it was in previous versions of the Ceph dashboard.

I am reaching out to inquire if there are any known issues, workarounds, or upcoming fixes for this. Your assistance in this matter would be greatly appreciated. Thank you for your time and support. I look forward to hearing from you soon. Best regards, Sanjay Mohan Software Defined Storage Engineer sanjaymohan@am.amrita.edu

[Amrita University] Disclaimer: The information transmitted in this email, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. Any views expressed in any message are those of the individual sender and may not necessarily reflect the views of Amrita Vishwa Vidyapeetham. If you received this in error, please contact the sender and destroy any copies of this information. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Frédéric, Thank you for the response. I tried disabling and re-enabling the module, and it seems that the recovery metrics are indeed being collected, but still not displayed on the new dashboard. Interestingly, I have another environment running Ceph 19.2.0 where the recovery throughput is displayed correctly, but I’m unable to identify any major differences between the two setups. Do you have any additional suggestions for troubleshooting this issue further? Thanks again for your help! Best regards, Sanjay Mohan Software Defined Storage Engineer sanjaymohan@am.amrita.edu ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: 21 October 2024 1:22 PM To: Sanjay Mohan <sanjaymohan@am.amrita.edu> Cc: ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Issue with Recovery Throughput Not Visible in Ceph Dashboard After Upgrade to 19.2.0 (Squid) Hi Sanjay, I've just checked the dashboard of a v19.2.0 cluster, and the recovery throughput is displayed correctly, as shown in the screenshot here [1]. You might want to consider redeploying the dashboard. Regards, Frédéric. [1] https://docs.ceph.com/en/latest/mgr/dashboard/ ----- Le 19 Oct 24, à 19:23, Sanjay Mohan sanjaymohan@am.amrita.edu a écrit :

Dear Ceph Users, I hope this message finds you well. I recently performed an upgrade of my Ceph cluster, moving through the following versions:

* 17.2.7 >> 18.2.0 >> 18.2.2 >> 18.2.4 >> 19.2.0 (Squid)

After successfully upgrading to Ceph 19.2.0, I noticed an issue where the recovery throughput is no longer visible in the new Ceph dashboard. However, Old dashboard metrics and features seem to be working as expected. It is important to note that the recovery throughput was displayed properly in the previous version of the Ceph dashboard. I am using Cephadm for the installation, not Rook. Current behavior:

* Recovery throughput metrics are not displayed in the new dashboard after upgrading to 19.2.0.

Expected behavior:

* The recovery throughput should be visible, as it was in previous versions of the Ceph dashboard.

I am reaching out to inquire if there are any known issues, workarounds, or upcoming fixes for this. Your assistance in this matter would be greatly appreciated. Thank you for your time and support. I look forward to hearing from you soon. Best regards, Sanjay Mohan Software Defined Storage Engineer sanjaymohan@am.amrita.edu

[Amrita University] Disclaimer: The information transmitted in this email, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. Any views expressed in any message are those of the individual sender and may not necessarily reflect the views of Amrita Vishwa Vidyapeetham. If you received this in error, please contact the sender and destroy any copies of this information. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io [Amrita University] Disclaimer: The information transmitted in this email, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. Any views expressed in any message are those of the individual sender and may not necessarily reflect the views of Amrita Vishwa Vidyapeetham. If you received this in error, please contact the sender and destroy any copies of this information.

Hey there, i've that problem too, although I got it from updating 17.2.7 to 18.2.4. After i read this mail I've just fiddled around a bit aaand Prometheus does not have ceph_osd_recovery_ops. Then i looked into the files in /var/lib/ceph/xyz/prometheus.node-name/etc/prometheus/prometheus.yml and there i found   - job_name: 'ceph-exporter'     honor_labels: true     http_sd_configs:     - url: http://132.252.90.151:8765/sd/prometheus/sd-config?service=ceph-exporter There are no ceph-exporter services running on my cluster, only node-exporters. So i started ceph-exporter services aaand there we can find ceph_osd_recovery_ops. I guess they got lost while upgrading? I dont know, but this seems to be the solution. ceph orch apply ceph-exporter ceph orch redeploy prometheus Best inDane On 21.10.24 13:59, Frédéric Nass wrote:

This could be due to a typo in the panel definition in Grafana (comparing the JSON of the working panel with the non-working one might provide more insights) or because the Prometheus Datasource used by Grafana isn't providing any metrics for ceph_osd_recovery_ops.

To check the panel in Grafana, you can go to https://<manager_ip>:3000 then Dashboard / Ceph - Cluster / Recovery Rate / Inspect JSON Panel. To verify if Prometheus provide some metrics, you can go to https://<manager_ip>:9095 and request metrics for ceph_osd_recovery_ops.

Check [1] if you need to set the Grafana password.

Regards, Frédéric

[1]https://www.ibm.com/docs/en/storage-ceph/6?topic=access-setting-admin-user-p...

----- Le 21 Oct 24, à 13:09, Sanjay Mohan<sanjaymohan@am.amrita.edu> a écrit :

...

...

Dear Ceph Users, I hope this message finds you well. I recently performed an upgrade of my Ceph cluster, moving through the following versions: * 17.2.7 >> 18.2.0 >> 18.2.2 >> 18.2.4 >> 19.2.0 (Squid) After successfully upgrading to Ceph 19.2.0, I noticed an issue where the recovery throughput is no longer visible in the new Ceph dashboard. However, Old dashboard metrics and features seem to be working as expected. It is important to note that the recovery throughput was displayed properly in the previous version of the Ceph dashboard. I am using Cephadm for the installation, not Rook. Current behavior: * Recovery throughput metrics are not displayed in the new dashboard after upgrading to 19.2.0. Expected behavior: * The recovery throughput should be visible, as it was in previous versions of the Ceph dashboard. I am reaching out to inquire if there are any known issues, workarounds, or upcoming fixes for this. Your assistance in this matter would be greatly appreciated. Thank you for your time and support. I look forward to hearing from you soon. Best regards, Sanjay Mohan Software Defined Storage Engineer sanjaymohan@am.amrita.edu [Amrita University] Disclaimer: The information transmitted in this email, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. Any views expressed in any message are those of the individual sender and may not necessarily reflect the views of Amrita Vishwa Vidyapeetham. If you received this in error, please contact the sender and destroy any copies of this information. _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io Disclaimer: The information transmitted in this email, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is

Hi Frédéric, Thank you for the response. I tried disabling and re-enabling the module, and it seems that the recovery metrics are indeed being collected, but still not displayed on the new dashboard. Interestingly, I have another environment running Ceph 19.2.0 where the recovery throughput is displayed correctly, but I’m unable to identify any major differences between the two setups. Do you have any additional suggestions for troubleshooting this issue further? Thanks again for your help! Best regards, Sanjay Mohan Software Defined Storage Engineer sanjaymohan@am.amrita.edu From: Frédéric Nass<frederic.nass@univ-lorraine.fr> Sent: 21 October 2024 1:22 PM To: Sanjay Mohan<sanjaymohan@am.amrita.edu> Cc: ceph-users<ceph-users@ceph.io> Subject: Re: [ceph-users] Issue with Recovery Throughput Not Visible in Ceph Dashboard After Upgrade to 19.2.0 (Squid) Hi Sanjay, I've just checked the dashboard of a v19.2.0 cluster, and the recovery throughput is displayed correctly, as shown in the screenshot here [1]. You might want to consider redeploying the dashboard. Regards, Frédéric. [1] [https://docs.ceph.com/en/latest/mgr/dashboard/ | https://docs.ceph.com/en/latest/mgr/dashboard/ ] ----- Le 19 Oct 24, à 19:23, Sanjay Mohansanjaymohan@am.amrita.edu a écrit : prohibited. Any views expressed in any message are those of the individual sender and may not necessarily reflect the views of Amrita Vishwa Vidyapeetham. If you received this in error, please contact the sender and destroy any copies of this information.

---

ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io

Hi everyone, I can confirm that both ceph-exporter and node-exporter are deployed in the non-upgraded v19 test cluster I have where recovery iops are visible in Ceph Dashboard. I find it odd that the dashboard requires an additional service to be deployed to display a single metric when all other metrics are displayed properly without it. Might be a transitionnal state with ceph-exporter being replaced by node-exporter and Prometheus configuration not being entirely updated yet. Just a guess, I haven't checked the tracker nor have I checked if node-exporter exports the recovery iops or not. Regards, Frédéric. ________________________________ De : Eugen Block <eblock@nde.ag> Envoyé : mardi 22 octobre 2024 11:20 À : ceph-users@ceph.io Objet : [ceph-users] Re: Issue with Recovery Throughput Not Visible in Ceph Dashboard After Upgrade to 19.2.0 (Squid) You're right, deploying ceph-exporter fixes the recovery graph in my  lab cluster. I didn't have to redeploy prometheus. Thanks! Zitat von mailing-lists <mailing-lists@indane.de>:

Hey there,

i've that problem too, although I got it from updating 17.2.7 to  18.2.4. After i read this mail I've just fiddled around a bit aaand  Prometheus does not have ceph_osd_recovery_ops.

Then i looked into the files in  /var/lib/ceph/xyz/prometheus.node-name/etc/prometheus/prometheus.yml  and there i found

  - job_name: 'ceph-exporter'     honor_labels: true     http_sd_configs:     - url:  http://132.252.90.151:8765/sd/prometheus/sd-config?service=ceph-exporter

There are no ceph-exporter services running on my cluster, only  node-exporters. So i started ceph-exporter services aaand there we  can find ceph_osd_recovery_ops.

I guess they got lost while upgrading? I dont know, but this seems  to be the solution.

ceph orch apply ceph-exporter

ceph orch redeploy prometheus

Best

inDane

On 21.10.24 13:59, Frédéric Nass wrote:

...

This could be due to a typo in the panel definition in Grafana  (comparing the JSON of the working panel with the non-working one  might provide more insights) or because the Prometheus Datasource  used by Grafana isn't providing any metrics for  ceph_osd_recovery_ops.

To check the panel in Grafana, you can go to  https://<manager_ip>:3000 then Dashboard / Ceph - Cluster /  Recovery Rate / Inspect JSON Panel. To verify if Prometheus provide some metrics, you can go to  https://<manager_ip>:9095 and request metrics for  ceph_osd_recovery_ops.

Check [1] if you need to set the Grafana password.

Regards, Frédéric

[1]https://www.ibm.com/docs/en/storage-ceph/6?topic=access-setting-admin-user-p... ----- Le 21 Oct 24, à 13:09, Sanjay Mohan<sanjaymohan@am.amrita.edu> a écrit  :

...

...

Dear Ceph Users, I hope this message finds you well. I recently performed an upgrade of my Ceph cluster, moving  through the following versions: * 17.2.7 >> 18.2.0 >> 18.2.2 >> 18.2.4 >> 19.2.0 (Squid) After successfully upgrading to Ceph 19.2.0, I noticed an issue where the recovery throughput is no longer visible in the new Ceph  dashboard. However, Old dashboard metrics and features seem to be working as expected. It is important to note that the recovery throughput was displayed  properly in the previous version of the Ceph dashboard. I am using Cephadm for the installation, not Rook. Current behavior: * Recovery throughput metrics are not displayed in the new dashboard after upgrading to 19.2.0. Expected behavior: * The recovery throughput should be visible, as it was in  previous versions of the Ceph dashboard. I am reaching out to inquire if there are any known issues,  workarounds, or upcoming fixes for this. Your assistance in this matter would be greatly appreciated. Thank you for your time and support. I look forward to hearing  from you soon. Best regards, Sanjay Mohan Software Defined Storage Engineer sanjaymohan@am.amrita.edu [Amrita University] Disclaimer: The information transmitted in this email, including  attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review,  retransmission, dissemination or other use of, or taking of any action in  reliance upon this information by persons or entities other than the intended recipient is prohibited. Any views expressed in any message are those of the individual sender and may not necessarily reflect the views of Amrita Vishwa  Vidyapeetham. If you received this in error, please contact the sender and  destroy any copies of this information. _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io Disclaimer: The information transmitted in this email, including  attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review,  retransmission, dissemination or other use of, or taking of any action in reliance  upon this information by persons or entities other than the intended recipient is

Hi Frédéric, Thank you for the response. I tried disabling and re-enabling the module, and it seems that  the recovery metrics are indeed being collected, but still not displayed on the new dashboard. Interestingly, I have another environment running Ceph  19.2.0 where the recovery throughput is displayed correctly, but I’m unable to  identify any major differences between the two setups. Do you have any additional suggestions for troubleshooting this  issue further? Thanks again for your help! Best regards, Sanjay Mohan Software Defined Storage Engineer sanjaymohan@am.amrita.edu From: Frédéric Nass<frederic.nass@univ-lorraine.fr> Sent: 21 October 2024 1:22 PM To: Sanjay Mohan<sanjaymohan@am.amrita.edu> Cc: ceph-users<ceph-users@ceph.io> Subject: Re: [ceph-users] Issue with Recovery Throughput Not  Visible in Ceph Dashboard After Upgrade to 19.2.0 (Squid) Hi Sanjay, I've just checked the dashboard of a v19.2.0 cluster, and the recovery throughput is displayed correctly, as shown in the screenshot here [1]. You might want to consider redeploying the dashboard. Regards, Frédéric. [1] [https://docs.ceph.com/en/latest/mgr/dashboard/ | https://docs.ceph.com/en/latest/mgr/dashboard/ ] ----- Le 19 Oct 24, à 19:23, Sanjay Mohansanjaymohan@am.amrita.edu  a écrit : prohibited. Any views expressed in any message are those of the individual sender and may not necessarily reflect the views of Amrita Vishwa  Vidyapeetham. If you received this in error, please contact the sender and  destroy any copies of this information.

---

ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io