2 Oct
2020
2 Oct
'20
5:25 p.m.
The important metric is the difference between these two values:
# ceph report | grep osdmap | grep committed
report 3324953770
"osdmap_first_committed": 3441952,
"osdmap_last_committed": 3442452,
The mon stores osdmaps on disk, and trims the older versions whenever
the PGs are clean. Trimming brings the osdmap_first_committed to be
closer to osdmap_last_committed.
In a cluster with no PGs backfilling or recovering, the mon should
trim that difference to be within 500-750 epochs.
If there are any PGs backfilling or recovering, then the mon will not
trim beyond the osdmap epoch when the pools were clean.
So if you are accumulating gigabytes of data in the mon dir, it
suggests that you have unclean PGs/Pools.
Cheers, dan
On Fri, Oct 2, 2020 at 4:14 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote:
>
>
> Does this also count if your cluster is not healthy because of errors
> like '2 pool(s) have no replicas configured'
> I sometimes use these pools for testing, they are empty.
>
>
>
>
> -----Original Message-----
> Cc: ceph-users
> Subject: [ceph-users] Re: Massive Mon DB Size with noout on 14.2.11
>
> As long as the cluster is no healthy, the OSD will require much more
> space, depending on the cluster size and other factors. Yes this is
> somewhat normal.
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
2 Oct
2 Oct
6:25 p.m.
New subject: Massive Mon DB Size with noout on 14.2.11
Hello *,
thx for taking care. I read "works as designed, be sure to have disk
space for the mon available". It sounds a little odd that the growth
from 50MB to ~15GB + compaction space happens within a couple of
seconds, when two OSD rejoin the cluster. Does it matter if I have
cephfs in use? Usually I would expect to have MDS load, but does it also
cause load on the mon with many files?
My OSD map seems to have low absolute numbers:
ceph report | grep osdmap | grep committed
report 777999536
"osdmap_first_committed": 1276,
"osdmap_last_committed": 1781,
If a get new disks (partitions) for the mons, is there a size
recommendation? Is there a rule of thumb? BTW: Do I still need a
filesystem for the partition of the mon DB?
Beste Regards,
derjohn
On 02.10.20 16:25, Dan van der Ster wrote:
The important metric is the difference between these
two values:
# ceph report | grep osdmap | grep committed
report 3324953770
"osdmap_first_committed": 3441952,
"osdmap_last_committed": 3442452,
The mon stores osdmaps on disk, and trims the older versions whenever
the PGs are clean. Trimming brings the osdmap_first_committed to be
closer to osdmap_last_committed.
In a cluster with no PGs backfilling or recovering, the mon should
trim that difference to be within 500-750 epochs.
If there are any PGs backfilling or recovering, then the mon will not
trim beyond the osdmap epoch when the pools were clean.
So if you are accumulating gigabytes of data in the mon dir, it
suggests that you have unclean PGs/Pools.
Cheers, dan
On Fri, Oct 2, 2020 at 4:14 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote:
--
Andreas John
net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach
Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832
Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net
Facebook: https://www.facebook.com/netlabdotnet
Twitter: https://twitter.com/netlabdotnet
Does this also count if your cluster is not healthy because of errors
like '2 pool(s) have no replicas configured'
I sometimes use these pools for testing, they are empty.
-----Original Message-----
Cc: ceph-users
Subject: [ceph-users] Re: Massive Mon DB Size with noout on 14.2.11
As long as the cluster is no healthy, the OSD will require much more
space, depending on the cluster size and other factors. Yes this is
somewhat normal.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:50 p.m.
New subject: Massive Mon DB Size with noout on 14.2.11
Hmm, in that case the osdmaps do not explain your high mon disk usage.
You'll have to investigate further...
-- dna
On Fri, Oct 2, 2020 at 5:26 PM Andreas John <aj(a)net-lab.net> wrote:
>
> Hello *,
>
> thx for taking care. I read "works as designed, be sure to have disk
> space for the mon available". It sounds a little odd that the growth
> from 50MB to ~15GB + compaction space happens within a couple of
> seconds, when two OSD rejoin the cluster. Does it matter if I have
> cephfs in use? Usually I would expect to have MDS load, but does it also
> cause load on the mon with many files?
>
> My OSD map seems to have low absolute numbers:
>
> ceph report | grep osdmap | grep committed
> report 777999536
> "osdmap_first_committed": 1276,
> "osdmap_last_committed": 1781,
>
>
> If a get new disks (partitions) for the mons, is there a size
> recommendation? Is there a rule of thumb? BTW: Do I still need a
> filesystem for the partition of the mon DB?
>
> Beste Regards,
>
> derjohn
>
>
> On 02.10.20 16:25, Dan van der Ster wrote:
> > The important metric is the difference between these two values:
> >
> > # ceph report | grep osdmap | grep committed
> > report 3324953770
> > "osdmap_first_committed": 3441952,
> > "osdmap_last_committed": 3442452,
> >
> > The mon stores osdmaps on disk, and trims the older versions whenever
> > the PGs are clean. Trimming brings the osdmap_first_committed to be
> > closer to osdmap_last_committed.
> > In a cluster with no PGs backfilling or recovering, the mon should
> > trim that difference to be within 500-750 epochs.
> >
> > If there are any PGs backfilling or recovering, then the mon will not
> > trim beyond the osdmap epoch when the pools were clean.
> >
> > So if you are accumulating gigabytes of data in the mon dir, it
> > suggests that you have unclean PGs/Pools.
> >
> > Cheers, dan
> >
> >
> >
> >
> > On Fri, Oct 2, 2020 at 4:14 PM Marc Roos <M.Roos(a)f1-outsourcing.eu>
wrote:
> >>
> >> Does this also count if your cluster is not healthy because of errors
> >> like '2 pool(s) have no replicas configured'
> >> I sometimes use these pools for testing, they are empty.
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> Cc: ceph-users
> >> Subject: [ceph-users] Re: Massive Mon DB Size with noout on 14.2.11
> >>
> >> As long as the cluster is no healthy, the OSD will require much more
> >> space, depending on the cluster size and other factors. Yes this is
> >> somewhat normal.
> >>
> >> _______________________________________________
> >> ceph-users mailing list -- ceph-users(a)ceph.io
> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
> --
> Andreas John
> net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach
> Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832
> Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net
>
> Facebook: https://www.facebook.com/netlabdotnet
> Twitter: https://twitter.com/netlabdotnet
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:05 p.m.
New subject: Massive Mon DB Size with noout on 14.2.11
thx for taking care. I read "works as designed,
be sure to have disk
space for the mon available”.
Well, yeah ;)
It sounds a little odd that the growth
from 50MB to ~15GB + compaction space happens within a couple of
seconds, when two OSD rejoin the cluster.
I’m suspicious — even on an Optane (which would be unusual for this application) it would
seem that writing 15GB -20GB (you mentioned both) of mon DB couldn’t happen within a
couple of seconds. Is it that `ceph -s` et al only start complaining when the DB reaches
that threshold, and that before the complaint it’s actually using like 14TB?
One thing, usually minor: dig into the mon DB directory. Are there extraneous files that
somehow are being dumped there? Next,
cd store.db ; ls -l LOG*
If there’s a huge LOG.old file, that can cause problems and can be deleted. If the LOG
file is huge, something else is going on.
If a get new disks (partitions) for the mons, is there
a size
recommendation? Is there a rule of thumb? BTW: Do I still need a
filesystem for the partition of the mon DB?
How small is your drive that you run out of space with just 20 GB of mon DB? Above you
hint that you have a separate filesystem for /var/lib/ceph — that could be your problem.
Did something dump a bunch of data in /var/tmp?
I tend to run mons with / using the whole of the drive (modulo /boot). So long as you
have proper rotation of /var/log/* configured in /etc/logrotate*, with decent size drives
you shouldn’t run out.
A pair of ~240GB SSDs usually suffices for mons; 400-480 are better yet.
6 Oct
6 Oct
1:52 p.m.
New subject: Massive Mon DB Size with noout on 14.2.11
Ok thanks, very clear, I am also indeed within this range.
-----Original Message-----
Subject: Re: [ceph-users] Re: Massive Mon DB Size with noout on 14.2.11
The important metric is the difference between these two values:
# ceph report | grep osdmap | grep committed report 3324953770
"osdmap_first_committed": 3441952,
"osdmap_last_committed": 3442452,
The mon stores osdmaps on disk, and trims the older versions whenever
the PGs are clean. Trimming brings the osdmap_first_committed to be
closer to osdmap_last_committed.
In a cluster with no PGs backfilling or recovering, the mon should trim
that difference to be within 500-750 epochs.
If there are any PGs backfilling or recovering, then the mon will not
trim beyond the osdmap epoch when the pools were clean.
So if you are accumulating gigabytes of data in the mon dir, it suggests
that you have unclean PGs/Pools.
Cheers, dan
On Fri, Oct 2, 2020 at 4:14 PM Marc Roos <M.Roos(a)f1-outsourcing.eu>
wrote:
Does this also count if your cluster is not healthy because of errors
like '2 pool(s) have no replicas configured'
I sometimes use these pools for testing, they are empty.
1306
days inactive
1310
days old
4 comments
4 participants
participants (4)
-
Andreas John -
Anthony D'Atri -
Dan van der Ster -
Marc Roos