Hello All, I'm hoping I can get some help with an issue in the dashboard after doing a recent bare metal ceph upgrade from Octopus to Quincy. ** Please note, this document references it only being an issue with the images tab shortly after this I found the same issue on another cluster that was recently upgraded from Octopus to quincy 17.2.7 within the last few months and it's affecting all tabs in the ceph dashboard it slows to a crawl until I restart or fail over the mgr both running on top of ubuntu 20.04 Everything appears to be working fine besides the Block --> images tab. It doesn't matter what node I fail over to, reboots, reinstalling ceph-mgr-dashboard, different broswers, clients etc It will not load the 4 RBDs I have, they appear in rbd ls, I can query them and the connection on the end appliance is fine. The loading icons spin infinitely without any failure message. If I access the images tab and then move to any other tab is the dashboard it will allow me to navigate but not display anything until I either restart the service on the active mgr or fail over to another, so it works as expected until I access this one tab. when I use any other section in the in the dashboard cpu utilization for the ceph-mgr is normal but when I access the images tab it's spiked to as high as 600% and will stay like that until I restart the service or fail over the active mgr -- Active MGR before clicking Block, the OSDs spike for a second but revert to around 5% top - 13:43:37 up 8 days, 23:09, 1 user, load average: 8.08, 5.02, 4.37 Tasks: 695 total, 1 running, 694 sleeping, 0 stopped, 0 zombie %Cpu(s): 7.0 us, 1.6 sy, 0.0 ni, 89.7 id, 1.2 wa, 0.0 hi, 0.5 si, 0.0 st ----- MiB Mem : 128474.1 total, 6705.6 free, 65684.0 used, 56084.5 buff/cache MiB Swap: 40927.0 total, 35839.3 free, 5087.7 used. 49253.0 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14156 ceph 20 0 3420632 1.9g 13668 S 55.3 1.5 864:49.51 ceph-osd 13762 ceph 20 0 3421384 1.8g 13432 S 51.3 1.4 960:22.12 ceph-osd 14163 ceph 20 0 3422352 1.7g 13016 S 50.0 1.3 902:41.19 ceph-osd 13803 ceph 20 0 3469596 1.8g 13532 S 44.7 1.4 941:55.10 ceph-osd 13774 ceph 20 0 3427560 1.7g 13656 S 38.7 1.4 932:02.51 ceph-osd 13801 ceph 20 0 3439796 1.7g 13448 S 37.7 1.3 981:25.55 ceph-osd 14025 ceph 20 0 3426360 1.8g 13780 S 36.4 1.4 994:00.75 ceph-osd 9888 nobody 20 0 126100 8696 0 S 21.2 0.0 1106:19 node_exporter 126798 ceph 20 0 1787824 528000 39464 S 7.9 0.4 0:14.84 ceph-mgr 13795 ceph 20 0 3420252 1.7g 13264 S 7.6 1.4 990:00.61 ceph-osd 13781 ceph 20 0 3484476 1.9g 13248 S 6.3 1.5 1040:10 ceph-osd 13777 ceph 20 0 3408972 1.8g 13464 S 6.0 1.5 1026:21 ceph-osd 13797 ceph 20 0 3432068 1.6g 13932 S 6.0 1.3 950:39.35 ceph-osd 13779 ceph 20 0 3471668 1.7g 12728 S 5.6 1.3 984:53.80 ceph-osd 13768 ceph 20 0 3496064 1.9g 13504 S 5.3 1.5 918:37.48 ceph-osd 13786 ceph 20 0 3422044 1.6g 13456 S 5.3 1.3 974:29.08 ceph-osd 13788 ceph 20 0 3454184 1.9g 13048 S 5.3 1.5 980:35.78 ceph-osd 13776 ceph 20 0 3445680 1.7g 12880 S 5.0 1.3 998:30.58 ceph-osd 13785 ceph 20 0 3409548 1.7g 13704 S 5.0 1.3 939:37.08 ceph-osd 14152 ceph 20 0 3465284 1.7g 13840 S 5.0 1.4 959:39.42 ceph-osd 10339 nobody 20 0 6256048 531428 60188 S 4.6 0.4 239:37.56 prometheus 13802 ceph 20 0 3430696 1.8g 13872 S 4.6 1.4 924:15.74 ceph-osd 13791 ceph 20 0 3498876 1.5g 12648 S 4.3 1.2 962:58.37 ceph-osd 13800 ceph 20 0 3455268 1.7g 12404 S 4.3 1.3 1000:41 ceph-osd 13790 ceph 20 0 3434364 1.6g 13516 S 3.3 1.3 974:16.46 ceph-osd 14217 ceph 20 0 3443436 1.8g 13560 S 3.3 1.4 902:54.22 ceph-osd 13526 ceph 20 0 1012048 499628 11244 S 3.0 0.4 349:35.28 ceph-mon 13775 ceph 20 0 3367284 1.6g 13940 S 3.0 1.3 878:38.27 ceph-osd 13784 ceph 20 0 3380960 1.8g 12892 S 3.0 1.4 910:50.47 ceph-osd 13789 ceph 20 0 3432876 1.6g 12464 S 2.6 1.2 922:45.15 ceph-osd 13804 ceph 20 0 3428120 1.9g 13192 S 2.6 1.5 865:31.30 ceph-osd 14153 ceph 20 0 3432752 1.8g 12576 S 2.3 1.4 874:27.92 ceph-osd 14192 ceph 20 0 3412640 1.9g 13512 S 2.3 1.5 923:01.97 ceph-osd 13796 ceph 20 0 3433016 1.8g 13164 S 2.0 1.4 982:08.21 ceph-osd 13798 ceph 20 0 3405708 1.6g 13508 S 2.0 1.3 873:50.34 ceph-osd 13814 ceph 20 0 4243252 1.5g 13500 S 2.0 1.2 2020:41 ceph-osd 13985 ceph 20 0 3487848 1.6g 13100 S 2.0 1.3 942:21.96 ceph-osd 14001 ceph 20 0 4194336 1.9g 13460 S 2.0 1.5 2143:46 ceph-osd 14186 ceph 20 0 3441852 1.5g 13360 S 2.0 1.2 956:30.81 ceph-osd 7257 root 20 0 82332 3480 2984 S 0.3 0.0 9:22.50 irqbalance 7269 syslog 20 0 224344 3648 2392 S 0.3 0.0 0:11.79 rsyslogd 16621 472 20 0 898376 79800 11080 S 0.3 0.1 146:36.08 grafana 104366 root 20 0 0 0 0 I 0.3 0.0 1:11.30 kworker/0:2- events 125676 root 20 0 0 0 0 I 0.3 0.0 0:00.48 kworker/u104:7- public-bond 127115 root 20 0 10172 4636 3392 R 0.3 0.0 0:00.10 top 1 root 20 0 180712 16312 5652 S 0.0 0.0 1:31.85 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.15 kthreadd --- top - 13:44:28 up 8 days, 23:09, 1 user, load average: 6.79, 5.11, 4.43 Tasks: 695 total, 1 running, 694 sleeping, 0 stopped, 0 zombie %Cpu(s): 12.9 us, 2.1 sy, 0.0 ni, 83.5 id, 1.0 wa, 0.0 hi, 0.5 si, 0.0 st MiB Mem : 128474.1 total, 6219.8 free, 66504.7 used, 55749.6 buff/cache MiB Swap: 40927.0 total, 35837.0 free, 5090.0 used. 48435.7 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 126798 ceph 20 0 1854596 573432 41484 S 482.5 0.4 1:45.70 ceph-mgr 14156 ceph 20 0 3420632 1.9g 13668 S 54.6 1.5 865:16.58 ceph-osd 13762 ceph 20 0 3421384 1.9g 13432 S 51.7 1.5 960:47.47 ceph-osd 13803 ceph 20 0 3469596 1.9g 13532 S 49.3 1.5 942:18.23 ceph-osd 14163 ceph 20 0 3422352 1.9g 13016 S 49.3 1.5 903:05.84 ceph-osd 13795 ceph 20 0 3420252 1.8g 13264 S 7.0 1.4 990:04.41 ceph-osd 13777 ceph 20 0 3408972 1.9g 13464 S 5.6 1.5 1026:24 ceph-osd 13786 ceph 20 0 3422044 1.6g 13456 S 5.3 1.3 974:32.09 ceph-osd 13797 ceph 20 0 3432068 1.6g 13932 S 5.3 1.3 950:42.02 ceph-osd 16621 472 20 0 898376 78776 11044 S 5.3 0.1 146:36.76 grafana 13791 ceph 20 0 3498876 1.5g 12648 S 5.0 1.2 963:00.94 ceph-osd 14001 ceph 20 0 4194336 1.9g 13460 S 5.0 1.5 2143:47 ceph-osd 9888 nobody 20 0 126100 8696 0 S 4.6 0.0 1106:23 node_exporter 13768 ceph 20 0 3496064 1.9g 13504 S 4.6 1.5 918:39.85 ceph-osd 13776 ceph 20 0 3445680 1.7g 12880 S 4.6 1.3 998:32.98 ceph-osd 13781 ceph 20 0 3484476 1.6g 13248 S 4.6 1.3 1040:13 ceph-osd 13785 ceph 20 0 3409548 1.7g 13704 S 4.6 1.3 939:40.08 ceph-osd 13788 ceph 20 0 3454184 1.9g 13048 S 4.6 1.5 980:38.24 ceph-osd 13779 ceph 20 0 3471668 1.7g 12728 S 4.3 1.3 984:56.39 ceph-osd 13800 ceph 20 0 3455268 1.7g 12404 S 4.3 1.3 1000:44 ceph-osd 13802 ceph 20 0 3430696 1.8g 13872 S 4.0 1.4 924:18.14 ceph-osd 14152 ceph 20 0 3465284 1.7g 13840 S 4.0 1.4 959:41.83 ceph-osd 13796 ceph 20 0 3433016 1.8g 13164 S 3.0 1.4 982:09.66 ceph-osd 13784 ceph 20 0 3380960 1.8g 12892 S 2.6 1.4 910:52.06 ceph-osd 13790 ceph 20 0 3434364 1.6g 13516 S 2.6 1.3 974:17.99 ceph-osd 13801 ceph 20 0 3439796 1.9g 13448 S 2.6 1.5 981:42.61 ceph-osd 14153 ceph 20 0 3432752 1.8g 12576 S 2.6 1.4 874:29.30 ceph-osd 14186 ceph 20 0 3441852 1.6g 13360 S 2.6 1.2 956:32.32 ceph-osd 14192 ceph 20 0 3412640 1.9g 13512 S 2.6 1.5 923:03.59 ceph-osd 13526 ceph 20 0 1012048 496332 11208 S 2.3 0.4 349:36.89 ceph-mon 13789 ceph 20 0 3432876 1.6g 12464 S 2.3 1.2 922:46.59 ceph-osd 13798 ceph 20 0 3405708 1.6g 13508 S 2.3 1.3 873:51.73 ceph-osd 14217 ceph 20 0 3443436 1.8g 13560 S 2.3 1.4 902:55.74 ceph-osd 13774 ceph 20 0 3427560 1.9g 13656 S 2.0 1.5 932:19.26 ceph-osd 13775 ceph 20 0 3367284 1.6g 13940 S 2.0 1.3 878:39.76 ceph-osd 13814 ceph 20 0 4243252 1.5g 13500 S 2.0 1.2 2020:42 ceph-osd 13985 ceph 20 0 3487848 1.6g 13100 S 2.0 1.3 942:23.39 ceph-osd 13804 ceph 20 0 3428120 1.9g 13192 S 1.7 1.5 865:32.71 ceph-osd 14025 ceph 20 0 3426360 1.8g 13780 S 1.7 1.5 994:17.32 ceph-osd 10339 nobody 20 0 6256048 537184 60136 S 1.0 0.4 239:38.44 prometheus 17547 nobody 20 0 128448 8572 0 S 1.0 0.0 31:54.99 alertmanager 127115 root 20 0 10172 4636 3392 R 0.7 0.0 0:00.43 top --- OS: Ubuntu 20.04 128GB of memory Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz X11SPL-F --- Ceph Versions { "mon": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4 }, "osd": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 140 }, "mds": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 4 }, "ctdb": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 1 }, "rgw": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 2 }, "overall": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 154 } } --- cluster: id: 388dda42-9dd0-4858-a978-b3dc4c3b9152 health: HEALTH_OK services: mon: 3 daemons, quorum jarn29,jarn30,jarn31 (age 8d) mgr: osd31(active, since 10m), standbys: osd30, osd29, osd32 mds: 2/2 daemons up, 2 standby osd: 140 osds: 140 up (since 3d), 140 in (since 3d) flags noautoscale ctdb: 1 daemon active (1 hosts) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 19 pools, 5024 pgs objects: 792.78M objects, 865 TiB usage: 1.4 PiB used, 586 TiB / 2.0 PiB avail pgs: 5013 active+clean 11 active+clean+scrubbing+deep io: client: 19 MiB/s rd, 102 MiB/s wr, 672 op/s rd, 351 op/s wr --- I've ran perf and included the ceph-mgr.log for the first system that is displaying the issue Perf - https://imgur.com/a/VMh4tDf mgr log while accessing RBD tab - https://pastebin.com/t96WCWfc mgr logs prior to clicking RBD -https://pastebin.com/e4dtuD3i --- Apologies for the formatting first time posting here If anything else is needed please let me know!