Greetings Josh, I executed the command today, and it effectively resolved the issue. Within moments, my pools became active, and read/write IOPS started to rise. Furthermore, the Hypervisor and VMs can now communicate seamlessly with the CEPH Cluster. *Command run:* ceph osd rm-pg-upmap-primary 21.a0 *To summarize our findings:* - Enabling the Ceph read balancer resulted in libvirtd from the hypervisor being unable to communicate with the CEPH cluster. - During the case, using RBD command images on the pool was readable I'd like to express my gratitude to everyone involved, especially the forum contributors. Regards Mosharaf Hossain Manager, Product Development IT Division BEXIMCO IT On Thu, Sep 14, 2023 at 7:58 PM Josh Salomon <jsalomon@redhat.com> wrote:
Hi Mosharaf - I will check it but I can assure that this error is a CLI error and the command has not impacted the system or the data. I have no clue what happened - I am sure I tested this scenario. The command syntax is ceph osd rm-pg-upmap-primary <PGID> the error you get is because you did not specify the pg id. Run this command in a loop for all pgs in a pool to return the pool to its original state, Regards,
Josh
On Thu, Sep 14, 2023 at 4:24 PM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Josh Thank you your for reply to us.
After giving the command in the cluster I got the following error. We are concerned about user data. Could you kindly confirm this command will not affect any user data?
root@ceph-node1:/# ceph osd rm-pg-upmap-primary Traceback (most recent call last): File "/usr/bin/ceph", line 1327, in <module> retval = main() File "/usr/bin/ceph", line 1036, in main retargs = run_in_thread(cluster_handle.conf_parse_argv, childargs) File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 1538, in run_in_thread raise t.exception File "/usr/lib/python3.6/site-packages/ceph_argparse.py", line 1504, in run self.retval = self.func(*self.args, **self.kwargs) File "rados.pyx", line 551, in rados.Rados.conf_parse_argv File "rados.pyx", line 314, in rados.cstr_list File "rados.pyx", line 308, in rados.cstr UnicodeEncodeError: 'utf-8' codec can't encode characters in position 3-4: surrogates
Apart from that, do you need any others information?
Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd.
On Thu, Sep 14, 2023 at 1:52 PM Josh Salomon <jsalomon@redhat.com> wrote:
Hi Mosharaf,
If you undo the read balancing commands (using the command 'ceph osd rm-pg-upmap-primary' on all pgs in the pool) do you see improvements in the performance?
Regards,
Josh
On Thu, Sep 14, 2023 at 12:35 AM Laura Flores <lflores@redhat.com> wrote:
Hi Mosharaf,
Can you please create a tracker issue and attach a copy of your osdmap? Also, please include any other output that characterizes the slowdown in client I/O operations you're noticing in your cluster. I can take a look once I have that information,
Thanks, Laura
On Wed, Sep 13, 2023 at 5:23 AM Mosharaf Hossain < mosharaf.hossain@bol-online.com> wrote:
Hello Folks We've recently performed an upgrade on our Cephadm cluster, transitioning from Ceph Quiency to Reef. However, following the manual implementation of a read balancer in the Reef cluster, we've experienced a significant slowdown in client I/O operations within the Ceph cluster, affecting both client bandwidth and overall cluster performance.
This slowdown has resulted in unresponsiveness across all virtual machines within the cluster, despite the fact that the cluster exclusively utilizes SSD storage."
Kindly guide us to move forward.
Regards Mosharaf Hossain _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804