Massive Mon DB Size with noout on 14.2.11
Hello, we observed massive and sudden growth of the mon db size on disk, from 50MB to 20GB+ (GB!) and thus reaching 100% disk usage on the mountpoint. As far as we can see, it happens if we set "noout" for a node reboot: After the node and the OSDs come back it looks like the mon db size increased drastically. We have 14.2.11, 10 OSD @ 2TB and cephfs in use. Is this a known issue? Should we avoid noout? TIA, derjohn -- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet
As long as the cluster is no healthy, the OSD will require much more space, depending on the cluster size and other factors. Yes this is somewhat normal. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am Fr., 2. Okt. 2020 um 15:46 Uhr schrieb Andreas John <aj@net-lab.net>:
Hello,
we observed massive and sudden growth of the mon db size on disk, from 50MB to 20GB+ (GB!) and thus reaching 100% disk usage on the mountpoint.
As far as we can see, it happens if we set "noout" for a node reboot: After the node and the OSDs come back it looks like the mon db size increased drastically.
We have 14.2.11, 10 OSD @ 2TB and cephfs in use.
Is this a known issue? Should we avoid noout?
TIA,
derjohn
-- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net
Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Does this also count if your cluster is not healthy because of errors like '2 pool(s) have no replicas configured' I sometimes use these pools for testing, they are empty. -----Original Message----- Cc: ceph-users Subject: [ceph-users] Re: Massive Mon DB Size with noout on 14.2.11 As long as the cluster is no healthy, the OSD will require much more space, depending on the cluster size and other factors. Yes this is somewhat normal.
One guy in the Russian Ceph chat had this problem when he had "insights" mgr module enabled. So try to disable various mgr modules and see if it helps...
Hello,
we observed massive and sudden growth of the mon db size on disk, from 50MB to 20GB+ (GB!) and thus reaching 100% disk usage on the mountpoint.
As far as we can see, it happens if we set "noout" for a node reboot: After the node and the OSDs come back it looks like the mon db size increased drastically.
We have 14.2.11, 10 OSD @ 2TB and cephfs in use.
Is this a known issue? Should we avoid noout?
TIA,
derjohn
-- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net
Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Andreas John
-
Marc Roos
-
Martin Verges
-
vitalif＠yourcmc.ru