22 Jan
2024
22 Jan
'24
11:56 a.m.
Hello,
last week I've got a HEALTH_OK on our CEPH cluster and I
started upgrade firmware in network cards.
When I had upgraded the sixth card from nine (one-by-one), this
server didn't started correctly and our ProxMox had problem with
accessing disk images on CEPH.
rbd ls pool
was OK, but:
rbd ls pool -l
didn't work. Our virtual servers had a trouble to work with
disks.
After I resolve network problem with OSD server, everythink
returning to normal state.
But I've found, that every OSD nod have very high activity: when
I've started 'iotop', there was very high load: around 180MB/s
read and 20MB/s write. In this time, cluster was in the HEALTH_OK
state. I've found, that there is a massive scrubbing activity...
After a few days, I have on our OSD nodes around 90MB/s read and
70MB/s write while 'ceph -s' have client io as 2,5MB/s read and
50MB/s write.
I've found in log file of our mon server many lines about
starting of scrubbing, but there are many messages about
starting of scrubb the same PG? I've grep'ed syslog for some of
them and attach it to this e-mail.
Is this activity OK? Why CEPH start scrubing this PG once and
once again?
And another question: Is scrubbing part of mClock scheduler?
Many thanks for explanation.
Sincerely
Jan Marek
--
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html
24 Jan
24 Jan
1:48 p.m.
[...] After a few days, I have on our OSD nodes around
90MB/s
read and 70MB/s write while 'ceph -s' have client io as
2,5MB/s read and 50MB/s write. [...]
This is one of my pet-peeves: that a storage system must have
capacity (principally IOPS) to handle both a maintenance
workload and a user workload, and since the former often
involves whole-storage or whole-metadata operations it can be
quite heavy, especially in the case of Ceph where rebalancing
and scrubbing and checking should be fairly frequent to detect
and correct inconsistencies.
Is this activity OK? [...]
Indeed. Some "clever" people "save money" by "rightsizing"
their
storage so it cannot run at the same time the maintenance and
the user workload, and so turn off the maintenance workload,
because they "feel lucky" I guess, but I do not recommend that.
:-). I have seen more than one Ceph cluster that did not have
the capacity even to run *just* the maintenance workload.
25 Jan
25 Jan
11:51 a.m.
Hello Peter,
your irony is perfect, it is worth to notice.
Meaning of my previous post was, that CEPH cluster didn't fulfill
my needs and, although I had set mClock profile to
"high_client_ops" (because I have a plenty of time to rebalancing
and scrubbing), my clients went to problems.
And there was question, if scheduler manage CEPH cluster
background (and clients) operation in this way to stil be usable
for clients.
I've tried to send feedback to developers.
Thanks for understanding.
Sincerely
Jan Marek
Dne St, led 24, 2024 at 11:18:20 CET napsal(a) Peter Grandi:
--
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html
[...] After a
few days, I have on our OSD nodes around 90MB/s
read and 70MB/s write while 'ceph -s' have client io as
2,5MB/s read and 50MB/s write. [...]
This is one of my pet-peeves: that a storage system must have
capacity (principally IOPS) to handle both a maintenance
workload and a user workload, and since the former often
involves whole-storage or whole-metadata operations it can be
quite heavy, especially in the case of Ceph where rebalancing
and scrubbing and checking should be fairly frequent to detect
and correct inconsistencies.
Is this activity OK? [...]
Indeed. Some "clever" people "save money" by "rightsizing"
their
storage so it cannot run at the same time the maintenance and
the user workload, and so turn off the maintenance workload,
because they "feel lucky" I guess, but I do not recommend that.
:-). I have seen more than one Ceph cluster that did not have
the capacity even to run *just* the maintenance workload.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
12:23 p.m.
Hello Jan,
Meaning of my previous post was, that CEPH cluster
didn't fulfill
my needs and, although I had set mClock profile to
"high_client_ops" (because I have a plenty of time to rebalancing
and scrubbing), my clients went to problems.
As far as the question around mClock is concerned, there are further
improvements in the works to handle QoS between client ops and
background scrub ops. This should help address the issue you are
currently facing. See PR: https://github.com/ceph/ceph/pull/51171
for more information.
Also, it would be helpful to know the Ceph version you are currently using.
-Sridhar
5:01 p.m.
Hello Sridhar,
Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
Hello Jan,
Meaning of my previous post was, that CEPH cluster didn't fulfill
my needs and, although I had set mClock profile to
"high_client_ops" (because I have a plenty of time to rebalancing
and scrubbing), my clients went to problems.
As far as the question around mClock is concerned, there are further
improvements in the works to handle QoS between client ops and
background scrub ops. This should help address the issue you are
currently facing. See PR: https://github.com/ceph/ceph/pull/51171
for more information.
Also, it would be helpful to know the Ceph version you are currently using.
thanks for your reply.
I've just in process upgrade between 17.2.6 and 18.2.1 (you can
see my previous posts about stuck in upgrade to reef).
Maybe this was cause of my problem...
Now I've tried give rest to the cluster to do some "background"
tasks (and it seems, that this was correct, because on my hosts
there is around 50-100MBps read and cca 10-50MBps write traffic -
cca 1/4-1/2 of previous load).
At Saturday I will change some settings on networking and I will
try to start upgrade process, maybe with --limit=1, to be "soft"
for cluster and for our clients...
-Sridhar
Sincerely
Jan Marek
--
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html
30 Jan
30 Jan
1:39 p.m.
Hello Sridhar,
at Saturday I've finished upgrade proces to 18.2.1.
Cluster is now in HEALTH_OK state and performs well.
According to my colleagues there are lower latences and good
throughput.
On OSD nodes there is relative low I/O activity.
I still have mClock profile "high_client_ops".
When I was stucked in the upgrade process, I had in logs so many
records, see attached file. Since upgrade is complete, this
messages went away... Can be this reason of poor
performance?
Sincerely
Jan Marek
Dne Čt, led 25, 2024 at 02:31:41 CET napsal(a) Jan Marek:
Hello Sridhar,
Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
Hello Jan,
Meaning of my previous post was, that CEPH cluster didn't fulfill
my needs and, although I had set mClock profile to
"high_client_ops" (because I have a plenty of time to rebalancing
and scrubbing), my clients went to problems.
As far as the question around mClock is concerned, there are further
improvements in the works to handle QoS between client ops and
background scrub ops. This should help address the issue you are
currently facing. See PR: https://github.com/ceph/ceph/pull/51171
for more information.
Also, it would be helpful to know the Ceph version you are currently using.
thanks for your reply.
I've just in process upgrade between 17.2.6 and 18.2.1 (you can
see my previous posts about stuck in upgrade to reef).
Maybe this was cause of my problem...
Now I've tried give rest to the cluster to do some "background"
tasks (and it seems, that this was correct, because on my hosts
there is around 50-100MBps read and cca 10-50MBps write traffic -
cca 1/4-1/2 of previous load).
At Saturday I will change some settings on networking and I will
try to start upgrade process, maybe with --limit=1, to be "soft"
for cluster and for our clients...
-Sridhar
Sincerely
Jan Marek
--
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html _______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html
1:46 p.m.
Hello again,
I'm sorry, I forgot attach file... :-(
Sincerely
Jan
Dne Út, led 30, 2024 at 11:09:44 CET napsal(a) Jan Marek:
Hello Sridhar,
at Saturday I've finished upgrade proces to 18.2.1.
Cluster is now in HEALTH_OK state and performs well.
According to my colleagues there are lower latences and good
throughput.
On OSD nodes there is relative low I/O activity.
I still have mClock profile "high_client_ops".
When I was stucked in the upgrade process, I had in logs so many
records, see attached file. Since upgrade is complete, this
messages went away... Can be this reason of poor
performance?
Sincerely
Jan Marek
Dne Čt, led 25, 2024 at 02:31:41 CET napsal(a) Jan Marek:
Hello Sridhar,
Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
Hello Jan,
Meaning of my previous post was, that CEPH cluster didn't fulfill
my needs and, although I had set mClock profile to
"high_client_ops" (because I have a plenty of time to rebalancing
and scrubbing), my clients went to problems.
As far as the question around mClock is concerned, there are further
improvements in the works to handle QoS between client ops and
background scrub ops. This should help address the issue you are
currently facing. See PR: https://github.com/ceph/ceph/pull/51171
for more information.
Also, it would be helpful to know the Ceph version you are currently using.
thanks for your reply.
I've just in process upgrade between 17.2.6 and 18.2.1 (you can
see my previous posts about stuck in upgrade to reef).
Maybe this was cause of my problem...
Now I've tried give rest to the cluster to do some "background"
tasks (and it seems, that this was correct, because on my hosts
there is around 50-100MBps read and cca 10-50MBps write traffic -
cca 1/4-1/2 of previous load).
At Saturday I will change some settings on networking and I will
try to start upgrade process, maybe with --limit=1, to be "soft"
for cluster and for our clients...
-Sridhar
Sincerely
Jan Marek
--
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html _______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html _______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
--
Ing. Jan Marek
University of South Bohemia
Academic Computer Centre
Phone: +420389032080
http://www.gnu.org/philosophy/no-word-attachments.cs.html
4:44 p.m.
Ah, yeah, you hit https://tracker.ceph.com/issues/63389 during the upgrade.
Josh
On Tue, Jan 30, 2024 at 3:17 AM Jan Marek <jmarek(a)jcu.cz> wrote:
>
> Hello again,
>
> I'm sorry, I forgot attach file... :-(
>
> Sincerely
> Jan
>
> Dne Út, led 30, 2024 at 11:09:44 CET napsal(a) Jan Marek:
> > Hello Sridhar,
> >
> > at Saturday I've finished upgrade proces to 18.2.1.
> >
> > Cluster is now in HEALTH_OK state and performs well.
> >
> > According to my colleagues there are lower latences and good
> > throughput.
> >
> > On OSD nodes there is relative low I/O activity.
> >
> > I still have mClock profile "high_client_ops".
> >
> > When I was stucked in the upgrade process, I had in logs so many
> > records, see attached file. Since upgrade is complete, this
> > messages went away... Can be this reason of poor
> > performance?
> >
> > Sincerely
> > Jan Marek
> >
> > Dne Čt, led 25, 2024 at 02:31:41 CET napsal(a) Jan Marek:
> > > Hello Sridhar,
> > >
> > > Dne Čt, led 25, 2024 at 09:53:26 CET napsal(a) Sridhar Seshasayee:
> > > > Hello Jan,
> > > >
> > > > Meaning of my previous post was, that CEPH cluster didn't fulfill
> > > > my needs and, although I had set mClock profile to
> > > > "high_client_ops" (because I have a plenty of time to
rebalancing
> > > > and scrubbing), my clients went to problems.
> > > >
> > > > As far as the question around mClock is concerned, there are further
> > > > improvements in the works to handle QoS between client ops and
> > > > background scrub ops. This should help address the issue you are
> > > > currently facing. See PR: https://github.com/ceph/ceph/pull/51171
> > > > for more information.
> > > > Also, it would be helpful to know the Ceph version you are currently
using.
> > >
> > > thanks for your reply.
> > >
> > > I've just in process upgrade between 17.2.6 and 18.2.1 (you can
> > > see my previous posts about stuck in upgrade to reef).
> > >
> > > Maybe this was cause of my problem...
> > >
> > > Now I've tried give rest to the cluster to do some
"background"
> > > tasks (and it seems, that this was correct, because on my hosts
> > > there is around 50-100MBps read and cca 10-50MBps write traffic -
> > > cca 1/4-1/2 of previous load).
> > >
> > > At Saturday I will change some settings on networking and I will
> > > try to start upgrade process, maybe with --limit=1, to be "soft"
> > > for cluster and for our clients...
> > >
> > > > -Sridhar
> > >
> > > Sincerely
> > > Jan Marek
> > > --
> > > Ing. Jan Marek
> > > University of South Bohemia
> > > Academic Computer Centre
> > > Phone: +420389032080
> > > http://www.gnu.org/philosophy/no-word-attachments.cs.html
> >
> >
> >
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users(a)ceph.io
> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> >
> > --
> > Ing. Jan Marek
> > University of South Bohemia
> > Academic Computer Centre
> > Phone: +420389032080
> > http://www.gnu.org/philosophy/no-word-attachments.cs.html
>
>
>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
> --
> Ing. Jan Marek
> University of South Bohemia
> Academic Computer Centre
> Phone: +420389032080
> http://www.gnu.org/philosophy/no-word-attachments.cs.html
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
102
days inactive
110
days old
7 comments
4 participants
participants (4)
-
Jan Marek -
Josh Baergen -
pg＠ceph.list.sabi.co.UK -
Sridhar Seshasayee