Hello, last week I've got a HEALTH_OK on our CEPH cluster and I started upgrade firmware in network cards. When I had upgraded the sixth card from nine (one-by-one), this server didn't started correctly and our ProxMox had problem with accessing disk images on CEPH. rbd ls pool was OK, but: rbd ls pool -l didn't work. Our virtual servers had a trouble to work with disks. After I resolve network problem with OSD server, everythink returning to normal state. But I've found, that every OSD nod have very high activity: when I've started 'iotop', there was very high load: around 180MB/s read and 20MB/s write. In this time, cluster was in the HEALTH_OK state. I've found, that there is a massive scrubbing activity... After a few days, I have on our OSD nodes around 90MB/s read and 70MB/s write while 'ceph -s' have client io as 2,5MB/s read and 50MB/s write. I've found in log file of our mon server many lines about starting of scrubbing, but there are many messages about starting of scrubb the same PG? I've grep'ed syslog for some of them and attach it to this e-mail. Is this activity OK? Why CEPH start scrubing this PG once and once again? And another question: Is scrubbing part of mClock scheduler? Many thanks for explanation. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
[...] After a few days, I have on our OSD nodes around 90MB/s read and 70MB/s write while 'ceph -s' have client io as 2,5MB/s read and 50MB/s write. [...]
This is one of my pet-peeves: that a storage system must have capacity (principally IOPS) to handle both a maintenance workload and a user workload, and since the former often involves whole-storage or whole-metadata operations it can be quite heavy, especially in the case of Ceph where rebalancing and scrubbing and checking should be fairly frequent to detect and correct inconsistencies.
Is this activity OK? [...]
Indeed. Some "clever" people "save money" by "rightsizing" their storage so it cannot run at the same time the maintenance and the user workload, and so turn off the maintenance workload, because they "feel lucky" I guess, but I do not recommend that. :-). I have seen more than one Ceph cluster that did not have the capacity even to run *just* the maintenance workload.
Hello Peter, your irony is perfect, it is worth to notice. Meaning of my previous post was, that CEPH cluster didn't fulfill my needs and, although I had set mClock profile to "high_client_ops" (because I have a plenty of time to rebalancing and scrubbing), my clients went to problems. And there was question, if scheduler manage CEPH cluster background (and clients) operation in this way to stil be usable for clients. I've tried to send feedback to developers. Thanks for understanding. Sincerely Jan Marek Dne St, led 24, 2024 at 11:18:20 CET napsal(a) Peter Grandi:
[...] After a few days, I have on our OSD nodes around 90MB/s read and 70MB/s write while 'ceph -s' have client io as 2,5MB/s read and 50MB/s write. [...]
This is one of my pet-peeves: that a storage system must have capacity (principally IOPS) to handle both a maintenance workload and a user workload, and since the former often involves whole-storage or whole-metadata operations it can be quite heavy, especially in the case of Ceph where rebalancing and scrubbing and checking should be fairly frequent to detect and correct inconsistencies.
Is this activity OK? [...]
Indeed. Some "clever" people "save money" by "rightsizing" their storage so it cannot run at the same time the maintenance and the user workload, and so turn off the maintenance workload, because they "feel lucky" I guess, but I do not recommend that. :-). I have seen more than one Ceph cluster that did not have the capacity even to run *just* the maintenance workload. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
Hello Jan,
Meaning of my previous post was, that CEPH cluster didn't fulfill my needs and, although I had set mClock profile to "high_client_ops" (because I have a plenty of time to rebalancing and scrubbing), my clients went to problems.
As far as the question around mClock is concerned, there are further improvements in the works to handle QoS between client ops and background scrub ops. This should help address the issue you are currently facing. See PR: https://github.com/ceph/ceph/pull/51171 for more information. Also, it would be helpful to know the Ceph version you are currently using. -Sridhar
Hello Sridhar, Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
Hello Jan,
Meaning of my previous post was, that CEPH cluster didn't fulfill my needs and, although I had set mClock profile to "high_client_ops" (because I have a plenty of time to rebalancing and scrubbing), my clients went to problems.
As far as the question around mClock is concerned, there are further improvements in the works to handle QoS between client ops and background scrub ops. This should help address the issue you are currently facing. See PR: https://github.com/ceph/ceph/pull/51171 for more information. Also, it would be helpful to know the Ceph version you are currently using.
thanks for your reply. I've just in process upgrade between 17.2.6 and 18.2.1 (you can see my previous posts about stuck in upgrade to reef). Maybe this was cause of my problem... Now I've tried give rest to the cluster to do some "background" tasks (and it seems, that this was correct, because on my hosts there is around 50-100MBps read and cca 10-50MBps write traffic - cca 1/4-1/2 of previous load). At Saturday I will change some settings on networking and I will try to start upgrade process, maybe with --limit=1, to be "soft" for cluster and for our clients...
-Sridhar
Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
Hello Sridhar, at Saturday I've finished upgrade proces to 18.2.1. Cluster is now in HEALTH_OK state and performs well. According to my colleagues there are lower latences and good throughput. On OSD nodes there is relative low I/O activity. I still have mClock profile "high_client_ops". When I was stucked in the upgrade process, I had in logs so many records, see attached file. Since upgrade is complete, this messages went away... Can be this reason of poor performance? Sincerely Jan Marek Dne Čt, led 25, 2024 at 02:31:41 CET napsal(a) Jan Marek:
Hello Sridhar,
Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
Hello Jan,
Meaning of my previous post was, that CEPH cluster didn't fulfill my needs and, although I had set mClock profile to "high_client_ops" (because I have a plenty of time to rebalancing and scrubbing), my clients went to problems.
As far as the question around mClock is concerned, there are further improvements in the works to handle QoS between client ops and background scrub ops. This should help address the issue you are currently facing. See PR: https://github.com/ceph/ceph/pull/51171 for more information. Also, it would be helpful to know the Ceph version you are currently using.
thanks for your reply.
I've just in process upgrade between 17.2.6 and 18.2.1 (you can see my previous posts about stuck in upgrade to reef).
Maybe this was cause of my problem...
Now I've tried give rest to the cluster to do some "background" tasks (and it seems, that this was correct, because on my hosts there is around 50-100MBps read and cca 10-50MBps write traffic - cca 1/4-1/2 of previous load).
At Saturday I will change some settings on networking and I will try to start upgrade process, maybe with --limit=1, to be "soft" for cluster and for our clients...
-Sridhar
Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
Hello again, I'm sorry, I forgot attach file... :-( Sincerely Jan Dne Út, led 30, 2024 at 11:09:44 CET napsal(a) Jan Marek:
Hello Sridhar,
at Saturday I've finished upgrade proces to 18.2.1.
Cluster is now in HEALTH_OK state and performs well.
According to my colleagues there are lower latences and good throughput.
On OSD nodes there is relative low I/O activity.
I still have mClock profile "high_client_ops".
When I was stucked in the upgrade process, I had in logs so many records, see attached file. Since upgrade is complete, this messages went away... Can be this reason of poor performance?
Sincerely Jan Marek
Dne Čt, led 25, 2024 at 02:31:41 CET napsal(a) Jan Marek:
Hello Sridhar,
Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
Hello Jan,
Meaning of my previous post was, that CEPH cluster didn't fulfill my needs and, although I had set mClock profile to "high_client_ops" (because I have a plenty of time to rebalancing and scrubbing), my clients went to problems.
As far as the question around mClock is concerned, there are further improvements in the works to handle QoS between client ops and background scrub ops. This should help address the issue you are currently facing. See PR: https://github.com/ceph/ceph/pull/51171 for more information. Also, it would be helpful to know the Ceph version you are currently using.
thanks for your reply.
I've just in process upgrade between 17.2.6 and 18.2.1 (you can see my previous posts about stuck in upgrade to reef).
Maybe this was cause of my problem...
Now I've tried give rest to the cluster to do some "background" tasks (and it seems, that this was correct, because on my hosts there is around 50-100MBps read and cca 10-50MBps write traffic - cca 1/4-1/2 of previous load).
At Saturday I will change some settings on networking and I will try to start upgrade process, maybe with --limit=1, to be "soft" for cluster and for our clients...
-Sridhar
Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
Ah, yeah, you hit https://tracker.ceph.com/issues/63389 during the upgrade. Josh On Tue, Jan 30, 2024 at 3:17 AM Jan Marek <jmarek@jcu.cz> wrote:
Hello again,
I'm sorry, I forgot attach file... :-(
Sincerely Jan
Dne Út, led 30, 2024 at 11:09:44 CET napsal(a) Jan Marek:
Hello Sridhar,
at Saturday I've finished upgrade proces to 18.2.1.
Cluster is now in HEALTH_OK state and performs well.
According to my colleagues there are lower latences and good throughput.
On OSD nodes there is relative low I/O activity.
I still have mClock profile "high_client_ops".
When I was stucked in the upgrade process, I had in logs so many records, see attached file. Since upgrade is complete, this messages went away... Can be this reason of poor performance?
Sincerely Jan Marek
Dne Čt, led 25, 2024 at 02:31:41 CET napsal(a) Jan Marek:
Hello Sridhar,
Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
Hello Jan,
Meaning of my previous post was, that CEPH cluster didn't fulfill my needs and, although I had set mClock profile to "high_client_ops" (because I have a plenty of time to rebalancing and scrubbing), my clients went to problems.
As far as the question around mClock is concerned, there are further improvements in the works to handle QoS between client ops and background scrub ops. This should help address the issue you are currently facing. See PR: https://github.com/ceph/ceph/pull/51171 for more information. Also, it would be helpful to know the Ceph version you are currently using.
thanks for your reply.
I've just in process upgrade between 17.2.6 and 18.2.1 (you can see my previous posts about stuck in upgrade to reef).
Maybe this was cause of my problem...
Now I've tried give rest to the cluster to do some "background" tasks (and it seems, that this was correct, because on my hosts there is around 50-100MBps read and cca 10-50MBps write traffic - cca 1/4-1/2 of previous load).
At Saturday I will change some settings on networking and I will try to start upgrade process, maybe with --limit=1, to be "soft" for cluster and for our clients...
-Sridhar
Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Jan Marek
-
Josh Baergen
-
pg＠ceph.list.sabi.co.UK
-
Sridhar Seshasayee