CEPH zero iops after upgrade to Reef and manual read balancer
Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance. This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage." Kindly guide us to move forward. Regards Mosharaf Hossain
Hi Mosharaf, Can you please create a tracker issue and attach a copy of your osdmap? Also, please include any other output that characterizes the slowdown in client I/O operations you're noticing in your cluster. I can take a look once I have that information, Thanks, Laura On Wed, Sep 13, 2023 at 5:23 AM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance.
This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage."
Kindly guide us to move forward.
Regards Mosharaf Hossain _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
Link the tracker on this list if you have it. You can create one under the RADOS project: https://tracker.ceph.com/projects/rados Thanks, Laura On Wed, Sep 13, 2023 at 4:35 PM Laura Flores <lflores@redhat.com> wrote:
Hi Mosharaf,
Can you please create a tracker issue and attach a copy of your osdmap? Also, please include any other output that characterizes the slowdown in client I/O operations you're noticing in your cluster. I can take a look once I have that information,
Thanks, Laura
On Wed, Sep 13, 2023 at 5:23 AM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance.
This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage."
Kindly guide us to move forward.
Regards Mosharaf Hossain _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
Hello Laura I have created the tracker and you can find it on https://tracker.ceph.com/issues/62836 Please find the OSD map as below on the cluster *root@cph2n1:/# ceph -s* cluster: id: e5f5ec6e-0b1b-11ec-adc5-35a84c0db1fb health: HEALTH_WARN 65 pgs not deep-scrubbed in time services: mon: 5 daemons, quorum cph2n1,cph2n2,cph2n3,cph2n5,cph2n4 (age 17h) mgr: cph2n6.kijcmg(active, since 17h), standbys: cph2n5.czgzpy, cph2n2.dqlmbn, cph2n4.gppcct, cph2n1.ezsddw osd: 56 osds: 56 up (since 8h), 56 in (since 8d) data: pools: 12 pools, 1913 pgs objects: 5.04M objects, 19 TiB usage: 57 TiB used, 98 TiB / 154 TiB avail pgs: 1913 active+clean io: client: 13 KiB/s rd, 2.8 MiB/s wr, 33 op/s rd, 44 op/s wr *root@cph2n1:/# ceph osd tree*ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 154.42514 root default -3 24.59732 host cph2n1 6 ent_ssd 3.49309 osd.6 up 1.00000 1.00000 7 ent_ssd 3.49309 osd.7 up 1.00000 1.00000 8 ent_ssd 3.49309 osd.8 up 1.00000 1.00000 9 ent_ssd 3.49309 osd.9 up 1.00000 1.00000 32 ent_ssd 3.49309 osd.32 up 1.00000 1.00000 37 ent_ssd 3.49309 osd.37 up 1.00000 1.00000 0 nvme_ssd 1.81940 osd.0 up 1.00000 1.00000 1 nvme_ssd 1.81940 osd.1 up 1.00000 1.00000 -5 24.59732 host cph2n2 10 ent_ssd 3.49309 osd.10 up 1.00000 1.00000 11 ent_ssd 3.49309 osd.11 up 1.00000 1.00000 12 ent_ssd 3.49309 osd.12 up 1.00000 1.00000 13 ent_ssd 3.49309 osd.13 up 1.00000 1.00000 31 ent_ssd 3.49309 osd.31 up 1.00000 1.00000 35 ent_ssd 3.49309 osd.35 up 1.00000 1.00000 2 nvme_ssd 1.81940 osd.2 up 1.00000 1.00000 3 nvme_ssd 1.81940 osd.3 up 1.00000 1.00000 -7 24.59732 host cph2n3 14 ent_ssd 3.49309 osd.14 up 1.00000 1.00000 15 ent_ssd 3.49309 osd.15 up 1.00000 1.00000 16 ent_ssd 3.49309 osd.16 up 1.00000 1.00000 17 ent_ssd 3.49309 osd.17 up 1.00000 1.00000 30 ent_ssd 3.49309 osd.30 up 1.00000 1.00000 36 ent_ssd 3.49309 osd.36 up 1.00000 1.00000 4 nvme_ssd 1.81940 osd.4 up 1.00000 1.00000 5 nvme_ssd 1.81940 osd.5 up 1.00000 1.00000 -21 28.09041 host cph2n4 22 ent_ssd 3.49309 osd.22 up 1.00000 1.00000 24 ent_ssd 3.49309 osd.24 up 1.00000 1.00000 26 ent_ssd 3.49309 osd.26 up 1.00000 1.00000 29 ent_ssd 3.49309 osd.29 up 1.00000 1.00000 33 ent_ssd 3.49309 osd.33 up 1.00000 1.00000 39 ent_ssd 3.49309 osd.39 up 1.00000 1.00000 41 ent_ssd 3.49309 osd.41 up 1.00000 1.00000 20 nvme_ssd 1.81940 osd.20 up 1.00000 1.00000 21 nvme_ssd 1.81940 osd.21 up 1.00000 1.00000 -17 28.09041 host cph2n5 23 ent_ssd 3.49309 osd.23 up 1.00000 1.00000 25 ent_ssd 3.49309 osd.25 up 1.00000 1.00000 27 ent_ssd 3.49309 osd.27 up 1.00000 1.00000 28 ent_ssd 3.49309 osd.28 up 1.00000 1.00000 34 ent_ssd 3.49309 osd.34 up 1.00000 1.00000 38 ent_ssd 3.49309 osd.38 up 1.00000 1.00000 40 ent_ssd 3.49309 osd.40 up 1.00000 1.00000 18 nvme_ssd 1.81940 osd.18 up 1.00000 1.00000 19 nvme_ssd 1.81940 osd.19 up 1.00000 1.00000 -25 12.22618 host cph2n6 42 ent_ssd 1.74660 osd.42 up 1.00000 1.00000 43 ent_ssd 1.74660 osd.43 up 1.00000 1.00000 44 ent_ssd 1.74660 osd.44 up 1.00000 1.00000 45 ent_ssd 1.74660 osd.45 up 1.00000 1.00000 49 ent_ssd 1.74660 osd.49 up 1.00000 1.00000 50 ent_ssd 1.74660 osd.50 up 1.00000 1.00000 51 ent_ssd 1.74660 osd.51 up 1.00000 1.00000 -29 12.22618 host cph2n7 52 ent_ssd 1.74660 osd.52 up 1.00000 1.00000 53 ent_ssd 1.74660 osd.53 up 1.00000 1.00000 54 ent_ssd 1.74660 osd.54 up 1.00000 1.00000 59 ent_ssd 1.74660 osd.59 up 1.00000 1.00000 60 ent_ssd 1.74660 osd.60 up 1.00000 1.00000 61 ent_ssd 1.74660 osd.61 up 1.00000 1.00000 62 ent_ssd 1.74660 osd.62 up 1.00000 1.00000 Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain@bol-online.com, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> On Thu, Sep 14, 2023 at 4:38 AM Laura Flores <lflores@redhat.com> wrote:
Link the tracker on this list if you have it. You can create one under the RADOS project: https://tracker.ceph.com/projects/rados
Thanks, Laura
On Wed, Sep 13, 2023 at 4:35 PM Laura Flores <lflores@redhat.com> wrote:
Hi Mosharaf,
Can you please create a tracker issue and attach a copy of your osdmap? Also, please include any other output that characterizes the slowdown in client I/O operations you're noticing in your cluster. I can take a look once I have that information,
Thanks, Laura
On Wed, Sep 13, 2023 at 5:23 AM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance.
This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage."
Kindly guide us to move forward.
Regards Mosharaf Hossain _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
Hi Mosharaf, If you undo the read balancing commands (using the command 'ceph osd rm-pg-upmap-primary' on all pgs in the pool) do you see improvements in the performance? Regards, Josh On Thu, Sep 14, 2023 at 12:35 AM Laura Flores <lflores@redhat.com> wrote:
Hi Mosharaf,
Can you please create a tracker issue and attach a copy of your osdmap? Also, please include any other output that characterizes the slowdown in client I/O operations you're noticing in your cluster. I can take a look once I have that information,
Thanks, Laura
On Wed, Sep 13, 2023 at 5:23 AM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance.
This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage."
Kindly guide us to move forward.
Regards Mosharaf Hossain _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
Hello Josh Thank you your for reply to us. After giving the command in the cluster I got the following error. We are concerned about user data. Could you kindly confirm this command will not affect any user data? root@ceph-node1:/# ceph osd rm-pg-upmap-primary Traceback (most recent call last): File "/usr/bin/ceph", line 1327, in <module> retval = main() File "/usr/bin/ceph", line 1036, in main retargs = run_in_thread(cluster_handle.conf_parse_argv, childargs) File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 1538, in run_in_thread raise t.exception File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 1504, in run self.retval = self.func(*self.args, **self.kwargs) File "rados.pyx", line 551, in rados.Rados.conf_parse_argv File "rados.pyx", line 314, in rados.cstr_list File "rados.pyx", line 308, in rados.cstr UnicodeEncodeError: 'utf-8' codec can't encode characters in position 3-4: surrogates Apart from that, do you need any others information? Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. On Thu, Sep 14, 2023 at 1:52 PM Josh Salomon <jsalomon@redhat.com> wrote:
Hi Mosharaf,
If you undo the read balancing commands (using the command 'ceph osd rm-pg-upmap-primary' on all pgs in the pool) do you see improvements in the performance?
Regards,
Josh
On Thu, Sep 14, 2023 at 12:35 AM Laura Flores <lflores@redhat.com> wrote:
Hi Mosharaf,
Can you please create a tracker issue and attach a copy of your osdmap? Also, please include any other output that characterizes the slowdown in client I/O operations you're noticing in your cluster. I can take a look once I have that information,
Thanks, Laura
On Wed, Sep 13, 2023 at 5:23 AM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance.
This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage."
Kindly guide us to move forward.
Regards Mosharaf Hossain _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
Hi Mosharaf - I will check it but I can assure that this error is a CLI error and the command has not impacted the system or the data. I have no clue what happened - I am sure I tested this scenario. The command syntax is ceph osd rm-pg-upmap-primary <PGID> the error you get is because you did not specify the pg id. Run this command in a loop for all pgs in a pool to return the pool to its original state, Regards, Josh On Thu, Sep 14, 2023 at 4:24 PM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Josh Thank you your for reply to us.
After giving the command in the cluster I got the following error. We are concerned about user data. Could you kindly confirm this command will not affect any user data?
root@ceph-node1:/# ceph osd rm-pg-upmap-primary Traceback (most recent call last): File "/usr/bin/ceph", line 1327, in <module> retval = main() File "/usr/bin/ceph", line 1036, in main retargs = run_in_thread(cluster_handle.conf_parse_argv, childargs) File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 1538, in run_in_thread raise t.exception File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 1504, in run self.retval = self.func(*self.args, **self.kwargs) File "rados.pyx", line 551, in rados.Rados.conf_parse_argv File "rados.pyx", line 314, in rados.cstr_list File "rados.pyx", line 308, in rados.cstr UnicodeEncodeError: 'utf-8' codec can't encode characters in position 3-4: surrogates
Apart from that, do you need any others information?
Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd.
On Thu, Sep 14, 2023 at 1:52 PM Josh Salomon <jsalomon@redhat.com> wrote:
Hi Mosharaf,
If you undo the read balancing commands (using the command 'ceph osd rm-pg-upmap-primary' on all pgs in the pool) do you see improvements in the performance?
Regards,
Josh
On Thu, Sep 14, 2023 at 12:35 AM Laura Flores <lflores@redhat.com> wrote:
Hi Mosharaf,
Can you please create a tracker issue and attach a copy of your osdmap? Also, please include any other output that characterizes the slowdown in client I/O operations you're noticing in your cluster. I can take a look once I have that information,
Thanks, Laura
On Wed, Sep 13, 2023 at 5:23 AM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance.
This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage."
Kindly guide us to move forward.
Regards Mosharaf Hossain _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
Greetings Josh, I executed the command today, and it effectively resolved the issue. Within moments, my pools became active, and read/write IOPS started to rise. Furthermore, the Hypervisor and VMs can now communicate seamlessly with the CEPH Cluster. *Command run:* ceph osd rm-pg-upmap-primary 21.a0 *To summarize our findings:* - Enabling the Ceph read balancer resulted in libvirtd from the hypervisor being unable to communicate with the CEPH cluster. - During the case, using RBD command images on the pool was readable I'd like to express my gratitude to everyone involved, especially the forum contributors. Regards Mosharaf Hossain Manager, Product Development IT Division BEXIMCO IT On Thu, Sep 14, 2023 at 7:58 PM Josh Salomon <jsalomon@redhat.com> wrote:
Hi Mosharaf - I will check it but I can assure that this error is a CLI error and the command has not impacted the system or the data. I have no clue what happened - I am sure I tested this scenario. The command syntax is ceph osd rm-pg-upmap-primary <PGID> the error you get is because you did not specify the pg id. Run this command in a loop for all pgs in a pool to return the pool to its original state, Regards,
Josh
On Thu, Sep 14, 2023 at 4:24 PM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Josh Thank you your for reply to us.
After giving the command in the cluster I got the following error. We are concerned about user data. Could you kindly confirm this command will not affect any user data?
root@ceph-node1:/# ceph osd rm-pg-upmap-primary Traceback (most recent call last): File "/usr/bin/ceph", line 1327, in <module> retval = main() File "/usr/bin/ceph", line 1036, in main retargs = run_in_thread(cluster_handle.conf_parse_argv, childargs) File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 1538, in run_in_thread raise t.exception File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 1504, in run self.retval = self.func(*self.args, **self.kwargs) File "rados.pyx", line 551, in rados.Rados.conf_parse_argv File "rados.pyx", line 314, in rados.cstr_list File "rados.pyx", line 308, in rados.cstr UnicodeEncodeError: 'utf-8' codec can't encode characters in position 3-4: surrogates
Apart from that, do you need any others information?
Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd.
On Thu, Sep 14, 2023 at 1:52 PM Josh Salomon <jsalomon@redhat.com> wrote:
Hi Mosharaf,
If you undo the read balancing commands (using the command 'ceph osd rm-pg-upmap-primary' on all pgs in the pool) do you see improvements in the performance?
Regards,
Josh
On Thu, Sep 14, 2023 at 12:35 AM Laura Flores <lflores@redhat.com> wrote:
Hi Mosharaf,
Can you please create a tracker issue and attach a copy of your osdmap? Also, please include any other output that characterizes the slowdown in client I/O operations you're noticing in your cluster. I can take a look once I have that information,
Thanks, Laura
On Wed, Sep 13, 2023 at 5:23 AM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance.
This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage."
Kindly guide us to move forward.
Regards Mosharaf Hossain _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
Hi Mosharaf, Thanks for the update. If you can reproduce the issue, it would be most helpful to us if you provided the output of `ceph -s`, along with a copy of your osdmap file. If you have this information, you can update the tracker here: https://tracker.ceph.com/issues/62836 Thanks, Laura On Mon, Sep 25, 2023 at 6:02 AM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Greetings Josh,
I executed the command today, and it effectively resolved the issue. Within moments, my pools became active, and read/write IOPS started to rise. Furthermore, the Hypervisor and VMs can now communicate seamlessly with the CEPH Cluster.
*Command run:* ceph osd rm-pg-upmap-primary 21.a0
*To summarize our findings:*
- Enabling the Ceph read balancer resulted in libvirtd from the hypervisor being unable to communicate with the CEPH cluster. - During the case, using RBD command images on the pool was readable
I'd like to express my gratitude to everyone involved, especially the forum contributors.
Regards Mosharaf Hossain Manager, Product Development IT Division BEXIMCO IT
On Thu, Sep 14, 2023 at 7:58 PM Josh Salomon <jsalomon@redhat.com> wrote:
Hi Mosharaf - I will check it but I can assure that this error is a CLI error and the command has not impacted the system or the data. I have no clue what happened - I am sure I tested this scenario. The command syntax is ceph osd rm-pg-upmap-primary <PGID> the error you get is because you did not specify the pg id. Run this command in a loop for all pgs in a pool to return the pool to its original state, Regards,
Josh
On Thu, Sep 14, 2023 at 4:24 PM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Josh Thank you your for reply to us.
After giving the command in the cluster I got the following error. We are concerned about user data. Could you kindly confirm this command will not affect any user data?
root@ceph-node1:/# ceph osd rm-pg-upmap-primary Traceback (most recent call last): File "/usr/bin/ceph", line 1327, in <module> retval = main() File "/usr/bin/ceph", line 1036, in main retargs = run_in_thread(cluster_handle.conf_parse_argv, childargs) File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 1538, in run_in_thread raise t.exception File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 1504, in run self.retval = self.func(*self.args, **self.kwargs) File "rados.pyx", line 551, in rados.Rados.conf_parse_argv File "rados.pyx", line 314, in rados.cstr_list File "rados.pyx", line 308, in rados.cstr UnicodeEncodeError: 'utf-8' codec can't encode characters in position 3-4: surrogates
Apart from that, do you need any others information?
Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd.
On Thu, Sep 14, 2023 at 1:52 PM Josh Salomon <jsalomon@redhat.com> wrote:
Hi Mosharaf,
If you undo the read balancing commands (using the command 'ceph osd rm-pg-upmap-primary' on all pgs in the pool) do you see improvements in the performance?
Regards,
Josh
On Thu, Sep 14, 2023 at 12:35 AM Laura Flores <lflores@redhat.com> wrote:
Hi Mosharaf,
Can you please create a tracker issue and attach a copy of your osdmap? Also, please include any other output that characterizes the slowdown in client I/O operations you're noticing in your cluster. I can take a look once I have that information,
Thanks, Laura
On Wed, Sep 13, 2023 at 5:23 AM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance.
This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage."
Kindly guide us to move forward.
Regards Mosharaf Hossain _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
participants (3)
-
Josh Salomon
-
Laura Flores
-
Mosharaf Hossain