29 Oct
2023
29 Oct
'23
8:33 p.m.
Hi all,
I see17.2.7 quincy is published as debian-bullseye packages. So I
tried it on a test cluster.
I must say I was not expecting the big dashboard change in a patch
release. Also all the "cluster utilization" numbers are all blank now
(any way to fix it?), so the dashboard is much less usable now.
Thoughts?
30 Oct
30 Oct
7:09 a.m.
Hi Matthew,
Is the prometheus configured in the cluster? And also the
PROMETHUEUS_API_URL is set? You can set it manually by ceph dashboard
set-prometheus-api-url <url-of-prom>.
You can switch to the old Dashboard by switching the feature toggle in the
dashboard. `ceph dashboard feature disable dashboard` and reloading the
page. Probably this should have been disabled by default.
Regards,
Nizam
On Sun, Oct 29, 2023, 23:04 Matthew Darwin <bugs(a)mdarwin.ca> wrote:
Hi all,
I see17.2.7 quincy is published as debian-bullseye packages. So I
tried it on a test cluster.
I must say I was not expecting the big dashboard change in a patch
release. Also all the "cluster utilization" numbers are all blank now
(any way to fix it?), so the dashboard is much less usable now.
Thoughts?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:38 p.m.
Hello,
We're not using prometheus within ceph (ceph dashboards show in our
grafana which is hosted elsewhere). The old dashboard showed the
metrics fine, so not sure why in a patch release we would need to make
configuration changes to get the same metrics.... Agree it should be
off by default.
"ceph dashboard feature disable dashboard" works to put the old
dashboard back. Thanks.
On 2023-10-30 00:09, Nizamudeen A wrote:
> Hi Matthew,
>
> Is the prometheus configured in the cluster? And also the
> PROMETHUEUS_API_URL is set? You can set it manually by ceph dashboard
> set-prometheus-api-url <url-of-prom>.
>
> You can switch to the old Dashboard by switching the feature toggle in the
> dashboard. `ceph dashboard feature disable dashboard` and reloading the
> page. Probably this should have been disabled by default.
>
> Regards,
> Nizam
>
> On Sun, Oct 29, 2023, 23:04 Matthew Darwin<bugs(a)mdarwin.ca> wrote:
>
>> Hi all,
>>
>> I see17.2.7 quincy is published as debian-bullseye packages. So I
>> tried it on a test cluster.
>>
>> I must say I was not expecting the big dashboard change in a patch
>> release. Also all the "cluster utilization" numbers are all blank now
>> (any way to fix it?), so the dashboard is much less usable now.
>>
>> Thoughts?
>> _______________________________________________
>> ceph-users mailing list --ceph-users(a)ceph.io
>> To unsubscribe send an email toceph-users-leave(a)ceph.io
>>
> _______________________________________________
> ceph-users mailing list --ceph-users(a)ceph.io
> To unsubscribe send an email toceph-users-leave(a)ceph.io
6:30 p.m.
Ah yeah, probably that's why the utilization charts are empty because it
relies on
the prometheus info.
And I raised a PR to disable the new dashboard in quincy.
https://github.com/ceph/ceph/pull/54250
Regards,
Nizam
On Mon, Oct 30, 2023 at 6:09 PM Matthew Darwin <bugs(a)mdarwin.ca> wrote:
Hello,
We're not using prometheus within ceph (ceph dashboards show in our
grafana which is hosted elsewhere). The old dashboard showed the
metrics fine, so not sure why in a patch release we would need to make
configuration changes to get the same metrics.... Agree it should be
off by default.
"ceph dashboard feature disable dashboard" works to put the old
dashboard back. Thanks.
On 2023-10-30 00:09, Nizamudeen A wrote:
Hi Matthew,
Is the prometheus configured in the cluster? And also the
PROMETHUEUS_API_URL is set? You can set it manually by ceph dashboard
set-prometheus-api-url <url-of-prom>.
You can switch to the old Dashboard by switching the feature toggle in
the
dashboard. `ceph dashboard feature disable
dashboard` and reloading the
page. Probably this should have been disabled by default.
Regards,
Nizam
On Sun, Oct 29, 2023, 23:04 Matthew Darwin<bugs(a)mdarwin.ca> wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hi all,
I see17.2.7 quincy is published as debian-bullseye packages. So I
tried it on a test cluster.
I must say I was not expecting the big dashboard change in a patch
release. Also all the "cluster utilization" numbers are all blank now
(any way to fix it?), so the dashboard is much less usable now.
Thoughts?
_______________________________________________
ceph-users mailing list --ceph-users(a)ceph.io
To unsubscribe send an email toceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list --ceph-users(a)ceph.io
To unsubscribe send an email toceph-users-leave(a)ceph.io
8:34 p.m.
New subject: 17.2.7 quincy dashboard issues
Ok, so I tried the new ceph dashboard by "set-prometheus-api-host"
(note "host" and not "url") and it returns the wrong data. We have 4
ceph clusters going into the same prometheus instance. How does it
know which data to pull? Do I need to pass a promql query?
The capacity widget at the top right (not using prometheus) shows 35%
of 51 TiB used (test cluster data)... This is correct. The chart shows
use capacity is 1.7 PiB, which is coming from the production cluster
(incorrect).
Ideas?
On 2023-10-30 11:30, Nizamudeen A wrote:
> Ah yeah, probably that's why the utilization charts are empty
> because it relies on
> the prometheus info.
>
> And I raised a PR to disable the new dashboard in quincy.
> https://github.com/ceph/ceph/pull/54250
>
> Regards,
> Nizam
>
> On Mon, Oct 30, 2023 at 6:09 PM Matthew Darwin <bugs(a)mdarwin.ca> wrote:
>
> Hello,
>
> We're not using prometheus within ceph (ceph dashboards show in our
> grafana which is hosted elsewhere). The old dashboard showed the
> metrics fine, so not sure why in a patch release we would need
> to make
> configuration changes to get the same metrics.... Agree it
> should be
> off by default.
>
> "ceph dashboard feature disable dashboard" works to put the old
> dashboard back. Thanks.
>
> On 2023-10-30 00:09, Nizamudeen A wrote:
> > Hi Matthew,
> >
> > Is the prometheus configured in the cluster? And also the
> > PROMETHUEUS_API_URL is set? You can set it manually by ceph
> dashboard
> > set-prometheus-api-url <url-of-prom>.
> >
> > You can switch to the old Dashboard by switching the feature
> toggle in the
> > dashboard. `ceph dashboard feature disable dashboard` and
> reloading the
> > page. Probably this should have been disabled by default.
> >
> > Regards,
> > Nizam
> >
> > On Sun, Oct 29, 2023, 23:04 Matthew Darwin<bugs(a)mdarwin.ca> wrote:
> >
> >> Hi all,
> >>
> >> I see17.2.7 quincy is published as debian-bullseye packages.
> So I
> >> tried it on a test cluster.
> >>
> >> I must say I was not expecting the big dashboard change in a
> patch
> >> release. Also all the "cluster utilization" numbers are all
> blank now
> >> (any way to fix it?), so the dashboard is much less usable now.
> >>
> >> Thoughts?
> >> _______________________________________________
> >> ceph-users mailing list --ceph-users(a)ceph.io
> >> To unsubscribe send an email toceph-users-leave(a)ceph.io
> >>
> > _______________________________________________
> > ceph-users mailing list --ceph-users(a)ceph.io
> > To unsubscribe send an email toceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
2 Nov
2 Nov
8:03 a.m.
New subject: 17.2.7 quincy dashboard issues
We have 4 ceph clusters going into the same prometheus instance.
Just curious, In the prometheus, if you want to see the details for a
single cluster, how's it done through query?
For reference, these are the queries that we are currently using now.
USEDCAPACITY = 'ceph_cluster_total_used_bytes',
WRITEIOPS = 'sum(rate(ceph_pool_wr[1m]))',
READIOPS = 'sum(rate(ceph_pool_rd[1m]))',
READLATENCY = 'avg_over_time(ceph_osd_apply_latency_ms[1m])',
WRITELATENCY = 'avg_over_time(ceph_osd_commit_latency_ms[1m])',
READCLIENTTHROUGHPUT = 'sum(rate(ceph_pool_rd_bytes[1m]))',
WRITECLIENTTHROUGHPUT = 'sum(rate(ceph_pool_wr_bytes[1m]))',
RECOVERYBYTES = 'sum(rate(ceph_osd_recovery_bytes[1m]))'
We might not have considered the possibility of multiple ceph-clusters
pointing to a single prometheus instance.
In that case there should be some filtering done with cluster id or
something to properly identify it.
FYI @Pedro Gonzalez Gomez <pegonzal(a)redhat.com> @Ankush Behl
<anbehl(a)redhat.com> @Aashish Sharma <aasharma(a)redhat.com>
Regards,
Nizam
On Mon, Oct 30, 2023 at 11:05 PM Matthew Darwin <bugs(a)mdarwin.ca> wrote:
Ok, so I tried the new ceph dashboard by
"set-prometheus-api-host"
(note "host" and not "url") and it returns the wrong data. We have
4
ceph clusters going into the same prometheus instance. How does it
know which data to pull? Do I need to pass a promql query?
The capacity widget at the top right (not using prometheus) shows 35%
of 51 TiB used (test cluster data)... This is correct. The chart shows
use capacity is 1.7 PiB, which is coming from the production cluster
(incorrect).
Ideas?
On 2023-10-30 11:30, Nizamudeen A wrote:
Ah yeah, probably that's why the utilization
charts are empty
because it relies on
the prometheus info.
And I raised a PR to disable the new dashboard in quincy.
https://github.com/ceph/ceph/pull/54250
Regards,
Nizam
On Mon, Oct 30, 2023 at 6:09 PM Matthew Darwin <bugs(a)mdarwin.ca> wrote:
Hello,
We're not using prometheus within ceph (ceph dashboards show in our
grafana which is hosted elsewhere). The old dashboard showed the
metrics fine, so not sure why in a patch release we would need
to make
configuration changes to get the same metrics.... Agree it
should be
off by default.
"ceph dashboard feature disable dashboard" works to put the old
dashboard back. Thanks.
On 2023-10-30 00:09, Nizamudeen A wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Hi Matthew,
Is the prometheus configured in the cluster? And also the
PROMETHUEUS_API_URL is set? You can set it manually by ceph
dashboard
set-prometheus-api-url <url-of-prom>.
You can switch to the old Dashboard by switching the feature
toggle in the
dashboard. `ceph dashboard feature disable
dashboard` and
reloading the
page. Probably this should have been disabled by
default.
Regards,
Nizam
On Sun, Oct 29, 2023, 23:04 Matthew Darwin<bugs(a)mdarwin.ca> wrote:
> Hi all,
>
> I see17.2.7 quincy is published as debian-bullseye packages.
So I
> tried it on a test cluster.
>
> I must say I was not expecting the big dashboard change in a
patch
> release. Also all the "cluster
utilization" numbers are all
blank now
(any way
to fix it?), so the dashboard is much less usable now.
Thoughts?
_______________________________________________
ceph-users mailing list --ceph-users(a)ceph.io
To unsubscribe send an email toceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list --ceph-users(a)ceph.io
To unsubscribe send an email toceph-users-leave(a)ceph.io
3 Nov
3 Nov
5:18 a.m.
New subject: 17.2.7 quincy dashboard issues
In my case I'm adding a label that is unique to each ceph cluster and
then can filter on that. In my ceph dashboard in grafana I've added a
pull-down list to check each different ceph cluster.
You need a way for me to configure what labels to filter on so I can
match it up with how I configured the prometheus. Alternately publish
all metrics to prometheus with fsid label then you can auto-filter
based on the fsid of the ceph cluster since fsid is unique.
On 2023-11-02 01:03, Nizamudeen A wrote:
>
> We have 4 ceph clusters going into the same prometheus instance.
>
> Just curious, In the prometheus, if you want to see the details for
> a single cluster, how's it done through query?
>
> For reference, these are the queries that we are currently using now.
>
> USEDCAPACITY = 'ceph_cluster_total_used_bytes',
> WRITEIOPS = 'sum(rate(ceph_pool_wr[1m]))',
> READIOPS = 'sum(rate(ceph_pool_rd[1m]))',
> READLATENCY = 'avg_over_time(ceph_osd_apply_latency_ms[1m])',
> WRITELATENCY = 'avg_over_time(ceph_osd_commit_latency_ms[1m])',
> READCLIENTTHROUGHPUT = 'sum(rate(ceph_pool_rd_bytes[1m]))',
> WRITECLIENTTHROUGHPUT = 'sum(rate(ceph_pool_wr_bytes[1m]))',
> RECOVERYBYTES = 'sum(rate(ceph_osd_recovery_bytes[1m]))'
>
> We might not have considered the possibility of multiple
> ceph-clusters pointing to a single prometheus instance.
> In that case there should be some filtering done with cluster id or
> something to properly identify it.
>
> FYI @Pedro Gonzalez Gomez <mailto:pegonzal@redhat.com> @Ankush Behl
> <mailto:anbehl@redhat.com> @Aashish Sharma <mailto:aasharma@redhat.com>
>
> Regards,
> Nizam
>
> On Mon, Oct 30, 2023 at 11:05 PM Matthew Darwin <bugs(a)mdarwin.ca> wrote:
>
> Ok, so I tried the new ceph dashboard by "set-prometheus-api-host"
> (note "host" and not "url") and it returns the wrong data.
We
> have 4
> ceph clusters going into the same prometheus instance. How does it
> know which data to pull? Do I need to pass a promql query?
>
> The capacity widget at the top right (not using prometheus)
> shows 35%
> of 51 TiB used (test cluster data)... This is correct. The chart
> shows
> use capacity is 1.7 PiB, which is coming from the production
> cluster
> (incorrect).
>
> Ideas?
>
>
> On 2023-10-30 11:30, Nizamudeen A wrote:
> > Ah yeah, probably that's why the utilization charts are empty
> > because it relies on
> > the prometheus info.
> >
> > And I raised a PR to disable the new dashboard in quincy.
> > https://github.com/ceph/ceph/pull/54250
> >
> > Regards,
> > Nizam
> >
> > On Mon, Oct 30, 2023 at 6:09 PM Matthew Darwin
> <bugs(a)mdarwin.ca> wrote:
> >
> > Hello,
> >
> > We're not using prometheus within ceph (ceph dashboards
> show in our
> > grafana which is hosted elsewhere). The old dashboard
> showed the
> > metrics fine, so not sure why in a patch release we would need
> > to make
> > configuration changes to get the same metrics.... Agree it
> > should be
> > off by default.
> >
> > "ceph dashboard feature disable dashboard" works to put
> the old
> > dashboard back. Thanks.
> >
> > On 2023-10-30 00:09, Nizamudeen A wrote:
> > > Hi Matthew,
> > >
> > > Is the prometheus configured in the cluster? And also the
> > > PROMETHUEUS_API_URL is set? You can set it manually by ceph
> > dashboard
> > > set-prometheus-api-url <url-of-prom>.
> > >
> > > You can switch to the old Dashboard by switching the feature
> > toggle in the
> > > dashboard. `ceph dashboard feature disable dashboard` and
> > reloading the
> > > page. Probably this should have been disabled by default.
> > >
> > > Regards,
> > > Nizam
> > >
> > > On Sun, Oct 29, 2023, 23:04 Matthew
> Darwin<bugs(a)mdarwin.ca> wrote:
> > >
> > >> Hi all,
> > >>
> > >> I see17.2.7 quincy is published as debian-bullseye
> packages.
> > So I
> > >> tried it on a test cluster.
> > >>
> > >> I must say I was not expecting the big dashboard change
> in a
> > patch
> > >> release. Also all the "cluster utilization" numbers
> are all
> > blank now
> > >> (any way to fix it?), so the dashboard is much less
> usable now.
> > >>
> > >> Thoughts?
> > >> _______________________________________________
> > >> ceph-users mailing list --ceph-users(a)ceph.io
> > >> To unsubscribe send an email toceph-users-leave(a)ceph.io
> > >>
> > > _______________________________________________
> > > ceph-users mailing list --ceph-users(a)ceph.io
> > > To unsubscribe send an email toceph-users-leave(a)ceph.io
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
9:42 a.m.
New subject: 17.2.7 quincy dashboard issues
Alternately publish
all metrics to prometheus with fsid label then you can auto-filter
based on the fsid of the ceph cluster since fsid is unique.
This is exactly something that we are looking into as we are looking into
providing
the support for multi-cluster monitoring and management from the Ceph
Dashboard
which is right now an ongoing PoC. Thanks for providing more context here.
But as of now, I don't see a way to make it configurable in the dashboard.
But in the near future
you can expect these to be added.
Regards,
Nizam
On Fri, Nov 3, 2023 at 7:54 AM Matthew Darwin <bugs(a)mdarwin.ca> wrote:
In my case I'm adding a label that is unique to
each ceph cluster and
then can filter on that. In my ceph dashboard in grafana I've added a
pull-down list to check each different ceph cluster.
You need a way for me to configure what labels to filter on so I can
match it up with how I configured the prometheus. Alternately publish
all metrics to prometheus with fsid label then you can auto-filter
based on the fsid of the ceph cluster since fsid is unique.
On 2023-11-02 01:03, Nizamudeen A wrote:
We have 4 ceph clusters going into the same prometheus instance.
Just curious, In the prometheus, if you want to see the details for
a single cluster, how's it done through query?
For reference, these are the queries that we are currently using now.
USEDCAPACITY = 'ceph_cluster_total_used_bytes',
WRITEIOPS = 'sum(rate(ceph_pool_wr[1m]))',
READIOPS = 'sum(rate(ceph_pool_rd[1m]))',
READLATENCY = 'avg_over_time(ceph_osd_apply_latency_ms[1m])',
WRITELATENCY = 'avg_over_time(ceph_osd_commit_latency_ms[1m])',
READCLIENTTHROUGHPUT = 'sum(rate(ceph_pool_rd_bytes[1m]))',
WRITECLIENTTHROUGHPUT = 'sum(rate(ceph_pool_wr_bytes[1m]))',
RECOVERYBYTES = 'sum(rate(ceph_osd_recovery_bytes[1m]))'
We might not have considered the possibility of multiple
ceph-clusters pointing to a single prometheus instance.
In that case there should be some filtering done with cluster id or
something to properly identify it.
FYI @Pedro Gonzalez Gomez <mailto:pegonzal@redhat.com> @Ankush Behl
<mailto:anbehl@redhat.com> @Aashish Sharma <mailto:aasharma@redhat.com>
Regards,
Nizam
On Mon, Oct 30, 2023 at 11:05 PM Matthew Darwin <bugs(a)mdarwin.ca> wrote:
Ok, so I tried the new ceph dashboard by "set-prometheus-api-host"
(note "host" and not "url") and it returns the wrong data. We
have 4
ceph clusters going into the same prometheus instance. How does it
know which data to pull? Do I need to pass a promql query?
The capacity widget at the top right (not using prometheus)
shows 35%
of 51 TiB used (test cluster data)... This is correct. The chart
shows
use capacity is 1.7 PiB, which is coming from the production
cluster
(incorrect).
Ideas?
On 2023-10-30 11:30, Nizamudeen A wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Ah yeah, probably that's why the utilization
charts are empty
because it relies on
the prometheus info.
And I raised a PR to disable the new dashboard in quincy.
https://github.com/ceph/ceph/pull/54250
Regards,
Nizam
On Mon, Oct 30, 2023 at 6:09 PM Matthew Darwin
<bugs(a)mdarwin.ca> wrote:
Hello,
We're not using prometheus within ceph (ceph dashboards
show in our
grafana which is hosted elsewhere). The old
dashboard
showed the
metrics fine, so not sure why in a patch
release we would need
to make
configuration changes to get the same metrics.... Agree it
should be
off by default.
"ceph dashboard feature disable dashboard" works to put
the old
dashboard back. Thanks.
On 2023-10-30 00:09, Nizamudeen A wrote:
Darwin<bugs(a)mdarwin.ca>
wrote:
Hi Matthew,
Is the prometheus configured in the cluster? And also the
PROMETHUEUS_API_URL is set? You can set it manually by ceph
dashboard
set-prometheus-api-url <url-of-prom>.
You can switch to the old Dashboard by switching the feature
toggle in the
dashboard. `ceph dashboard feature disable
dashboard` and
reloading the
> page. Probably this should have been disabled by default.
>
> Regards,
> Nizam
>
> On Sun, Oct 29, 2023, 23:04 Matthew >
>> Hi all,
>>
>> I see17.2.7 quincy is published as debian-bullseye
packages.
So I
>> tried it on a test cluster.
>>
>> I must say I was not expecting the big dashboard change
in a
patch
>> release. Also all the "cluster utilization" numbers
are all
blank now
>> (any way to fix it?), so the dashboard is much less
usable now.
>
> Thoughts?
> _______________________________________________
> ceph-users mailing list --ceph-users(a)ceph.io
> To unsubscribe send an email toceph-users-leave(a)ceph.io
>
_______________________________________________
ceph-users mailing list --ceph-users(a)ceph.io
To unsubscribe send an email toceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
199
days inactive
204
days old
7 comments
2 participants
participants (2)
-
Matthew Darwin -
Nizamudeen A