Hi all, I have a fairly pressing issue. I had a monitor fall out of quorum because it ran out of disk space during rebalancing from switching to upmap. I noticed all my monitor store.db started taking up nearly all disk space so I set noout, nobackfill and norecover and shutdown all the monitor daemons. Each store.db was at: mon.a 89GB (the one that firt dropped out) mon.a 400GB mon.c 400GB I tried setting mon_compact_on_start. This brought mon.a down to 1GB. Cool. However, when I try it on the other monitors it increased the db size ~1Gb/10s so I shut them down again. Any idea what is going on? Or how can I shrik back down the db?
Hi Ricardo, I just had a similar issue recently. I did a dump of the monitor store (i.e., something like "ceph-monstore-tool /var/lib/ceph/mon/mon-a/ dump-keys") and most messages were of type 'logm'. For me I think it was a lot of log messages coming from an oddly behaving OSD. I've seen folks advise disabling the Ceph mgr insights module if you have it running and there are degraded PGs, to see if that helps. What finally solved it for me was doing a rolling restart of my nodes, but I started from all PGs active+clean. --Lincoln ________________________________ From: ricardo.re.azevedo@gmail.com <ricardo.re.azevedo@gmail.com> Sent: Wednesday, March 10, 2021 5:59 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] mon db growing. over 500Gb Hi all, I have a fairly pressing issue. I had a monitor fall out of quorum because it ran out of disk space during rebalancing from switching to upmap. I noticed all my monitor store.db started taking up nearly all disk space so I set noout, nobackfill and norecover and shutdown all the monitor daemons. Each store.db was at: mon.a 89GB (the one that firt dropped out) mon.a 400GB mon.c 400GB I tried setting mon_compact_on_start. This brought mon.a down to 1GB. Cool. However, when I try it on the other monitors it increased the db size ~1Gb/10s so I shut them down again. Any idea what is going on? Or how can I shrik back down the db? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the input Lincoln. I think I am in a similar boat. I don't have the insight module activated. I checked one of my troublesome monitors with the command you game and indeed it is full of logm messages. I am not sure what would have caused it though. My OSDs have been behaving relatively ok. I tried rebooting all my OSDs with mds and mgr disabled and am in the same spot. Starting the mon manually with mon_compact_on_start gives: ceph@a /m/c/m/c/store.db> /usr/bin/ceph-mon -f --cluster ceph --id a --setuser ceph --setgroup ceph ignoring --setuser ceph since I am not root ignoring --setgroup ceph since I am not root 2021-03-10T16:47:10.017-0800 7f076f143540 -1 compacting monitor store ... It hangs on compaction meanwhile the store.db keeps expanding. Seems like there is something wrong with compaction since I don't think the mon is connected yet at this point and I have every other ceph service disabled. I had to increase ulimit at this point. any thoughts on how to procede? Is there a way I can clear the db of these messages? Thanks everyone From: Lincoln Bryant <lincolnb@uchicago.edu> Sent: Wednesday, March 10, 2021 4:06 PM To: ricardo.re.azevedo@gmail.com; ceph-users@ceph.io Subject: Re: [ceph-users] mon db growing. over 500Gb Hi Ricardo, I just had a similar issue recently. I did a dump of the monitor store (i.e., something like "ceph-monstore-tool /var/lib/ceph/mon/mon-a/ dump-keys") and most messages were of type 'logm'. For me I think it was a lot of log messages coming from an oddly behaving OSD. I've seen folks advise disabling the Ceph mgr insights module if you have it running and there are degraded PGs, to see if that helps. What finally solved it for me was doing a rolling restart of my nodes, but I started from all PGs active+clean. --Lincoln _____ From: ricardo.re.azevedo@gmail.com <mailto:ricardo.re.azevedo@gmail.com> <ricardo.re.azevedo@gmail.com <mailto:ricardo.re.azevedo@gmail.com> > Sent: Wednesday, March 10, 2021 5:59 PM To: ceph-users@ceph.io <mailto:ceph-users@ceph.io> <ceph-users@ceph.io <mailto:ceph-users@ceph.io> > Subject: [ceph-users] mon db growing. over 500Gb Hi all, I have a fairly pressing issue. I had a monitor fall out of quorum because it ran out of disk space during rebalancing from switching to upmap. I noticed all my monitor store.db started taking up nearly all disk space so I set noout, nobackfill and norecover and shutdown all the monitor daemons. Each store.db was at: mon.a 89GB (the one that firt dropped out) mon.a 400GB mon.c 400GB I tried setting mon_compact_on_start. This brought mon.a down to 1GB. Cool. However, when I try it on the other monitors it increased the db size ~1Gb/10s so I shut them down again. Any idea what is going on? Or how can I shrik back down the db? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
You can try compacting with monstore tool instead of using mon-compact-on-start. I am not sure if it makes any difference. ________________________________ From: ricardo.re.azevedo@gmail.com <ricardo.re.azevedo@gmail.com> Sent: Wednesday, March 10, 2021 6:59 PM To: Lincoln Bryant <lincolnb@uchicago.edu>; ceph-users@ceph.io <ceph-users@ceph.io> Subject: RE: [ceph-users] mon db growing. over 500Gb Thanks for the input Lincoln. I think I am in a similar boat. I don’t have the insight module activated. I checked one of my troublesome monitors with the command you game and indeed it is full of logm messages. I am not sure what would have caused it though. My OSDs have been behaving relatively ok. I tried rebooting all my OSDs with mds and mgr disabled and am in the same spot. Starting the mon manually with mon_compact_on_start gives: ceph@a /m/c/m/c/store.db> /usr/bin/ceph-mon -f --cluster ceph --id a --setuser ceph --setgroup ceph ignoring --setuser ceph since I am not root ignoring --setgroup ceph since I am not root 2021-03-10T16:47:10.017-0800 7f076f143540 -1 compacting monitor store ... It hangs on compaction meanwhile the store.db keeps expanding. Seems like there is something wrong with compaction since I don’t think the mon is connected yet at this point and I have every other ceph service disabled. I had to increase ulimit at this point. any thoughts on how to procede? Is there a way I can clear the db of these messages? Thanks everyone From: Lincoln Bryant <lincolnb@uchicago.edu> Sent: Wednesday, March 10, 2021 4:06 PM To: ricardo.re.azevedo@gmail.com; ceph-users@ceph.io Subject: Re: [ceph-users] mon db growing. over 500Gb Hi Ricardo, I just had a similar issue recently. I did a dump of the monitor store (i.e., something like "ceph-monstore-tool /var/lib/ceph/mon/mon-a/ dump-keys") and most messages were of type 'logm'. For me I think it was a lot of log messages coming from an oddly behaving OSD. I've seen folks advise disabling the Ceph mgr insights module if you have it running and there are degraded PGs, to see if that helps. What finally solved it for me was doing a rolling restart of my nodes, but I started from all PGs active+clean. --Lincoln ________________________________ From: ricardo.re.azevedo@gmail.com<mailto:ricardo.re.azevedo@gmail.com> <ricardo.re.azevedo@gmail.com<mailto:ricardo.re.azevedo@gmail.com>> Sent: Wednesday, March 10, 2021 5:59 PM To: ceph-users@ceph.io<mailto:ceph-users@ceph.io> <ceph-users@ceph.io<mailto:ceph-users@ceph.io>> Subject: [ceph-users] mon db growing. over 500Gb Hi all, I have a fairly pressing issue. I had a monitor fall out of quorum because it ran out of disk space during rebalancing from switching to upmap. I noticed all my monitor store.db started taking up nearly all disk space so I set noout, nobackfill and norecover and shutdown all the monitor daemons. Each store.db was at: mon.a 89GB (the one that firt dropped out) mon.a 400GB mon.c 400GB I tried setting mon_compact_on_start. This brought mon.a down to 1GB. Cool. However, when I try it on the other monitors it increased the db size ~1Gb/10s so I shut them down again. Any idea what is going on? Or how can I shrik back down the db? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
From what I have read here in the past, growing monitor db is related to not having pg's in 'clean active' state
-----Original Message----- From: ricardo.re.azevedo@gmail.com <ricardo.re.azevedo@gmail.com> Sent: 11 March 2021 00:59 To: ceph-users@ceph.io Subject: [ceph-users] mon db growing. over 500Gb
Hi all,
I have a fairly pressing issue. I had a monitor fall out of quorum because it ran out of disk space during rebalancing from switching to upmap. I noticed all my monitor store.db started taking up nearly all disk space so I set noout, nobackfill and norecover and shutdown all the monitor daemons. Each store.db was at:
mon.a 89GB (the one that firt dropped out)
mon.a 400GB
mon.c 400GB
I tried setting mon_compact_on_start. This brought mon.a down to 1GB. Cool. However, when I try it on the other monitors it increased the db size ~1Gb/10s so I shut them down again.
Any idea what is going on? Or how can I shrik back down the db?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, I also observed excessively growing mon DB in case of recovery. Luckily we were able to solve it by exdending the mon db disk. Without having the chance to re-check: The options nobackfill and norecover might cause that behavior.It feelds like mon holds data that cannot be flushed to an OSD. rgds, j. On 11.03.21 10:47, Marc wrote:
From what I have read here in the past, growing monitor db is related to not having pg's in 'clean active' state
-----Original Message----- From: ricardo.re.azevedo@gmail.com <ricardo.re.azevedo@gmail.com> Sent: 11 March 2021 00:59 To: ceph-users@ceph.io Subject: [ceph-users] mon db growing. over 500Gb
Hi all,
I have a fairly pressing issue. I had a monitor fall out of quorum because it ran out of disk space during rebalancing from switching to upmap. I noticed all my monitor store.db started taking up nearly all disk space so I set noout, nobackfill and norecover and shutdown all the monitor daemons. Each store.db was at:
mon.a 89GB (the one that firt dropped out)
mon.a 400GB
mon.c 400GB
I tried setting mon_compact_on_start. This brought mon.a down to 1GB. Cool. However, when I try it on the other monitors it increased the db size ~1Gb/10s so I shut them down again.
Any idea what is going on? Or how can I shrik back down the db?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet
HI Andreas, That's good to know. I managed to fix the problem! Here is my journey in case it helps anyone: My system drives are only 512GB so I added spare 1Tb drives to each server and moved the mon db to the new drive. I set noout, nobackfill and norecover and enabled only the ceph mon and osd services (disabled mgr and mds in case they were throwing the log messages). I then let it sit. In the first hour the db expanded: mon.a: 1GB ->80GB mon.b: 500GB ->550GB mon.c: 500GB ->500GB then after another hour mon.a increased to 100GB but mon.c dropped to 50GB. After another hour mon.a and mon.c were down to ~10Gb. By the next morning the final mon was also ~10Gb and the cluster was happy again. Thank you ceph! It would be great to know what caused this initial inflation but my take away is to keep the mon db on a drive separate the OS in case of db overinflation (and the 10GB min hardware requirements should have an asterisk if this is a common issue). I think part of my issue was that inflation started interfering with OS functions, exacerbating things. Thanks all for your help. Definitely helped me sort things out. Best, Ricardo -----Original Message----- From: Andreas John <aj@net-lab.net> Sent: Thursday, March 11, 2021 2:32 AM To: ceph-users@ceph.io Subject: [ceph-users] Re: mon db growing. over 500Gb Hello, I also observed excessively growing mon DB in case of recovery. Luckily we were able to solve it by exdending the mon db disk. Without having the chance to re-check: The options nobackfill and norecover might cause that behavior.It feelds like mon holds data that cannot be flushed to an OSD. rgds, j. On 11.03.21 10:47, Marc wrote:
From what I have read here in the past, growing monitor db is related to not having pg's in 'clean active' state
-----Original Message----- From: ricardo.re.azevedo@gmail.com <ricardo.re.azevedo@gmail.com> Sent: 11 March 2021 00:59 To: ceph-users@ceph.io Subject: [ceph-users] mon db growing. over 500Gb
Hi all,
I have a fairly pressing issue. I had a monitor fall out of quorum because it ran out of disk space during rebalancing from switching to upmap. I noticed all my monitor store.db started taking up nearly all disk space so I set noout, nobackfill and norecover and shutdown all the monitor daemons. Each store.db was at:
mon.a 89GB (the one that firt dropped out)
mon.a 400GB
mon.c 400GB
I tried setting mon_compact_on_start. This brought mon.a down to 1GB. Cool. However, when I try it on the other monitors it increased the db size ~1Gb/10s so I shut them down again.
Any idea what is going on? Or how can I shrik back down the db?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Andreas John
-
Lincoln Bryant
-
Marc
-
ricardo.re.azevedo＠gmail.com