Hello Laura I have created the tracker and you can find it on https://tracker.ceph.com/issues/62836 Please find the OSD map as below on the cluster *root@cph2n1:/# ceph -s* cluster: id: e5f5ec6e-0b1b-11ec-adc5-35a84c0db1fb health: HEALTH_WARN 65 pgs not deep-scrubbed in time services: mon: 5 daemons, quorum cph2n1,cph2n2,cph2n3,cph2n5,cph2n4 (age 17h) mgr: cph2n6.kijcmg(active, since 17h), standbys: cph2n5.czgzpy, cph2n2.dqlmbn, cph2n4.gppcct, cph2n1.ezsddw osd: 56 osds: 56 up (since 8h), 56 in (since 8d) data: pools: 12 pools, 1913 pgs objects: 5.04M objects, 19 TiB usage: 57 TiB used, 98 TiB / 154 TiB avail pgs: 1913 active+clean io: client: 13 KiB/s rd, 2.8 MiB/s wr, 33 op/s rd, 44 op/s wr *root@cph2n1:/# ceph osd tree*ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 154.42514 root default -3 24.59732 host cph2n1 6 ent_ssd 3.49309 osd.6 up 1.00000 1.00000 7 ent_ssd 3.49309 osd.7 up 1.00000 1.00000 8 ent_ssd 3.49309 osd.8 up 1.00000 1.00000 9 ent_ssd 3.49309 osd.9 up 1.00000 1.00000 32 ent_ssd 3.49309 osd.32 up 1.00000 1.00000 37 ent_ssd 3.49309 osd.37 up 1.00000 1.00000 0 nvme_ssd 1.81940 osd.0 up 1.00000 1.00000 1 nvme_ssd 1.81940 osd.1 up 1.00000 1.00000 -5 24.59732 host cph2n2 10 ent_ssd 3.49309 osd.10 up 1.00000 1.00000 11 ent_ssd 3.49309 osd.11 up 1.00000 1.00000 12 ent_ssd 3.49309 osd.12 up 1.00000 1.00000 13 ent_ssd 3.49309 osd.13 up 1.00000 1.00000 31 ent_ssd 3.49309 osd.31 up 1.00000 1.00000 35 ent_ssd 3.49309 osd.35 up 1.00000 1.00000 2 nvme_ssd 1.81940 osd.2 up 1.00000 1.00000 3 nvme_ssd 1.81940 osd.3 up 1.00000 1.00000 -7 24.59732 host cph2n3 14 ent_ssd 3.49309 osd.14 up 1.00000 1.00000 15 ent_ssd 3.49309 osd.15 up 1.00000 1.00000 16 ent_ssd 3.49309 osd.16 up 1.00000 1.00000 17 ent_ssd 3.49309 osd.17 up 1.00000 1.00000 30 ent_ssd 3.49309 osd.30 up 1.00000 1.00000 36 ent_ssd 3.49309 osd.36 up 1.00000 1.00000 4 nvme_ssd 1.81940 osd.4 up 1.00000 1.00000 5 nvme_ssd 1.81940 osd.5 up 1.00000 1.00000 -21 28.09041 host cph2n4 22 ent_ssd 3.49309 osd.22 up 1.00000 1.00000 24 ent_ssd 3.49309 osd.24 up 1.00000 1.00000 26 ent_ssd 3.49309 osd.26 up 1.00000 1.00000 29 ent_ssd 3.49309 osd.29 up 1.00000 1.00000 33 ent_ssd 3.49309 osd.33 up 1.00000 1.00000 39 ent_ssd 3.49309 osd.39 up 1.00000 1.00000 41 ent_ssd 3.49309 osd.41 up 1.00000 1.00000 20 nvme_ssd 1.81940 osd.20 up 1.00000 1.00000 21 nvme_ssd 1.81940 osd.21 up 1.00000 1.00000 -17 28.09041 host cph2n5 23 ent_ssd 3.49309 osd.23 up 1.00000 1.00000 25 ent_ssd 3.49309 osd.25 up 1.00000 1.00000 27 ent_ssd 3.49309 osd.27 up 1.00000 1.00000 28 ent_ssd 3.49309 osd.28 up 1.00000 1.00000 34 ent_ssd 3.49309 osd.34 up 1.00000 1.00000 38 ent_ssd 3.49309 osd.38 up 1.00000 1.00000 40 ent_ssd 3.49309 osd.40 up 1.00000 1.00000 18 nvme_ssd 1.81940 osd.18 up 1.00000 1.00000 19 nvme_ssd 1.81940 osd.19 up 1.00000 1.00000 -25 12.22618 host cph2n6 42 ent_ssd 1.74660 osd.42 up 1.00000 1.00000 43 ent_ssd 1.74660 osd.43 up 1.00000 1.00000 44 ent_ssd 1.74660 osd.44 up 1.00000 1.00000 45 ent_ssd 1.74660 osd.45 up 1.00000 1.00000 49 ent_ssd 1.74660 osd.49 up 1.00000 1.00000 50 ent_ssd 1.74660 osd.50 up 1.00000 1.00000 51 ent_ssd 1.74660 osd.51 up 1.00000 1.00000 -29 12.22618 host cph2n7 52 ent_ssd 1.74660 osd.52 up 1.00000 1.00000 53 ent_ssd 1.74660 osd.53 up 1.00000 1.00000 54 ent_ssd 1.74660 osd.54 up 1.00000 1.00000 59 ent_ssd 1.74660 osd.59 up 1.00000 1.00000 60 ent_ssd 1.74660 osd.60 up 1.00000 1.00000 61 ent_ssd 1.74660 osd.61 up 1.00000 1.00000 62 ent_ssd 1.74660 osd.62 up 1.00000 1.00000 Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> On Thu, Sep 14, 2023 at 4:38 AM Laura Flores <lflores@redhat.com> wrote:
Link the tracker on this list if you have it. You can create one under the RADOS project: https://tracker.ceph.com/projects/rados
Thanks, Laura
On Wed, Sep 13, 2023 at 4:35 PM Laura Flores <lflores@redhat.com> wrote:
Hi Mosharaf,
Can you please create a tracker issue and attach a copy of your osdmap? Also, please include any other output that characterizes the slowdown in client I/O operations you're noticing in your cluster. I can take a look once I have that information,
Thanks, Laura
On Wed, Sep 13, 2023 at 5:23 AM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance.
This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage."
Kindly guide us to move forward.
Regards Mosharaf Hossain _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804