Yes, I have, Eugen, I see no obvious reason / error / etc. I see a lot of entries relating to Compressing as well as monitor going down. Andrei ----- Original Message ----- > Have you looked into syslog and mon logs? > > > Zitat von Andrei Mikhailovsky <andrei(a)arhont.com>om>: > >> Hello everyone, >> >> I am having regular messages that the Monitors are going down and up: >> >> 2020-10-27T09:50:49.032431+0000 mon .arh-ibstorage2-ib ( mon .1) >> 2248 : cluster [WRN] Health check failed: 1/4 mons down, quorum >> arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib (MON_DOWN) >> 2020-10-27T09:50:49.123511+0000 mon .arh-ibstorage2-ib ( mon .1) >> 2250 : cluster [WRN] overall HEALTH_WARN 23 OSD(s) experiencing >> BlueFS spillover; 3 large omap objects; 1/4 mons down, quorum >> arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib; noout flag(s) >> set; 43 pgs not deep-scrubbed in time; 12 pgs not scrubbed in time >> 2020-10-27T09:50:52.735457+0000 mon .arh-ibstorage1-ib ( mon .0) >> 31287 : cluster [INF] Health check cleared: MON_DOWN (was: 1/4 mons >> down, quorum arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib) >> 2020-10-27T12:35:20.556458+0000 mon .arh-ibstorage2-ib ( mon .1) >> 2260 : cluster [WRN] Health check failed: 1/4 mons down, quorum >> arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib (MON_DOWN) >> 2020-10-27T12:35:20.643282+0000 mon .arh-ibstorage2-ib ( mon .1) >> 2262 : cluster [WRN] overall HEALTH_WARN 23 OSD(s) experiencing >> BlueFS spillover; 3 large omap objects; 1/4 mons down, quorum >> arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib; noout flag(s) >> set; 47 pgs not deep-scrubbed in time; 14 pgs not scrubbed in time >> >> >> This happens on a daily basis several times a day. >> >> Could you please let me know how to fix this annoying problem? >> >> I am running ceph version 15.2.4 >> (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) on >> Ubuntu 18.04 LTS with latest updates. >> >> Thanks >> >> Andrei >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

Eugen, I've got four physical servers and I've installed mon on all of them. I've discussed it with Wido and a few other chaps from ceph and there is no issue in doing it. The quorum issues would happen if you have 2 mons. If you've got more than 2 you should be fine. Andrei ----- Original Message ----- > Why do you have 4 MONs in the first place? That way a quorum is > difficult to achieve, could it be related to that? > > Zitat von Andrei Mikhailovsky <andrei(a)arhont.com>om>: > >> Yes, I have, Eugen, I see no obvious reason / error / etc. I see a >> lot of entries relating to Compressing as well as monitor going down. >> >> Andrei >> >> >> >> ----- Original Message ----- >>> From: "Eugen Block" <eblock(a)nde.ag> >>> To: "ceph-users" <ceph-users(a)ceph.io> >>> Sent: Wednesday, 28 October, 2020 11:51:20 >>> Subject: [ceph-users] Re: frequent Monitor down >> >>> Have you looked into syslog and mon logs? >>> >>> >>> Zitat von Andrei Mikhailovsky <andrei(a)arhont.com>om>: >>> >>>> Hello everyone, >>>> >>>> I am having regular messages that the Monitors are going down and up: >>>> >>>> 2020-10-27T09:50:49.032431+0000 mon .arh-ibstorage2-ib ( mon .1) >>>> 2248 : cluster [WRN] Health check failed: 1/4 mons down, quorum >>>> arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib (MON_DOWN) >>>> 2020-10-27T09:50:49.123511+0000 mon .arh-ibstorage2-ib ( mon .1) >>>> 2250 : cluster [WRN] overall HEALTH_WARN 23 OSD(s) experiencing >>>> BlueFS spillover; 3 large omap objects; 1/4 mons down, quorum >>>> arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib; noout flag(s) >>>> set; 43 pgs not deep-scrubbed in time; 12 pgs not scrubbed in time >>>> 2020-10-27T09:50:52.735457+0000 mon .arh-ibstorage1-ib ( mon .0) >>>> 31287 : cluster [INF] Health check cleared: MON_DOWN (was: 1/4 mons >>>> down, quorum arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib) >>>> 2020-10-27T12:35:20.556458+0000 mon .arh-ibstorage2-ib ( mon .1) >>>> 2260 : cluster [WRN] Health check failed: 1/4 mons down, quorum >>>> arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib (MON_DOWN) >>>> 2020-10-27T12:35:20.643282+0000 mon .arh-ibstorage2-ib ( mon .1) >>>> 2262 : cluster [WRN] overall HEALTH_WARN 23 OSD(s) experiencing >>>> BlueFS spillover; 3 large omap objects; 1/4 mons down, quorum >>>> arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib; noout flag(s) >>>> set; 47 pgs not deep-scrubbed in time; 14 pgs not scrubbed in time >>>> >>>> >>>> This happens on a daily basis several times a day. >>>> >>>> Could you please let me know how to fix this annoying problem? >>>> >>>> I am running ceph version 15.2.4 >>>> (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) on >>>> Ubuntu 18.04 LTS with latest updates. >>>> >>>> Thanks >>>> >>>> Andrei >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

Typically, the number of nodes is 2n+1 to cover n failures. It's OK to have 4 nodes, from failure covering POV, it's the same as 3 nodes. 4 nodes will cover 1 failure. If 2 nodes down, the cluster is down. It works, just not make much sense. Thanks! Tony > -----Original Message----- > From: Marc Roos <M.Roos(a)f1-outsourcing.eu> > Sent: Thursday, October 29, 2020 1:42 AM > To: andrei <andrei(a)arhont.com>om>; eblock <eblock(a)nde.ag> > Cc: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: frequent Monitor down > > Really? First time I read this here, afaik you can get a split brain > like this. > > > > -----Original Message----- > Sent: Thursday, October 29, 2020 12:16 AM > To: Eugen Block > Cc: ceph-users > Subject: [ceph-users] Re: frequent Monitor down > > Eugen, I've got four physical servers and I've installed mon on all of > them. I've discussed it with Wido and a few other chaps from ceph and > there is no issue in doing it. The quorum issues would happen if you > have 2 mons. If you've got more than 2 you should be fine. > > Andrei > > ----- Original Message ----- > > From: "Eugen Block" <eblock(a)nde.ag> > > To: "Andrei Mikhailovsky" <andrei(a)arhont.com> > > Cc: "ceph-users" <ceph-users(a)ceph.io> > > Sent: Wednesday, 28 October, 2020 20:19:15 > > Subject: Re: [ceph-users] Re: frequent Monitor down > > > Why do you have 4 MONs in the first place? That way a quorum is > > difficult to achieve, could it be related to that? > > > > Zitat von Andrei Mikhailovsky <andrei(a)arhont.com>om>: > > > >> Yes, I have, Eugen, I see no obvious reason / error / etc. I see a > >> lot of entries relating to Compressing as well as monitor going down. > >> > >> Andrei > >> > >> > >> > >> ----- Original Message ----- > >>> From: "Eugen Block" <eblock(a)nde.ag> > >>> To: "ceph-users" <ceph-users(a)ceph.io> > >>> Sent: Wednesday, 28 October, 2020 11:51:20 > >>> Subject: [ceph-users] Re: frequent Monitor down > >> > >>> Have you looked into syslog and mon logs? > >>> > >>> > >>> Zitat von Andrei Mikhailovsky <andrei(a)arhont.com>om>: > >>> > >>>> Hello everyone, > >>>> > >>>> I am having regular messages that the Monitors are going down and > up: > >>>> > >>>> 2020-10-27T09:50:49.032431+0000 mon .arh-ibstorage2-ib ( mon .1) > >>>> 2248 : cluster [WRN] Health check failed: 1/4 mons down, quorum > >>>> arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib (MON_DOWN) > >>>> 2020-10-27T09:50:49.123511+0000 mon .arh-ibstorage2-ib ( mon .1) > >>>> 2250 : cluster [WRN] overall HEALTH_WARN 23 OSD(s) experiencing > >>>> BlueFS spillover; 3 large omap objects; 1/4 mons down, quorum > >>>> arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib; noout > >>>> flag(s) set; 43 pgs not deep-scrubbed in time; 12 pgs not scrubbed > >>>> in time 2020-10-27T09:50:52.735457+0000 mon .arh-ibstorage1-ib ( > >>>> mon .0) > >>>> 31287 : cluster [INF] Health check cleared: MON_DOWN (was: 1/4 mons > > >>>> down, quorum arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib) > >>>> 2020-10-27T12:35:20.556458+0000 mon .arh-ibstorage2-ib ( mon .1) > >>>> 2260 : cluster [WRN] Health check failed: 1/4 mons down, quorum > >>>> arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib (MON_DOWN) > >>>> 2020-10-27T12:35:20.643282+0000 mon .arh-ibstorage2-ib ( mon .1) > >>>> 2262 : cluster [WRN] overall HEALTH_WARN 23 OSD(s) experiencing > >>>> BlueFS spillover; 3 large omap objects; 1/4 mons down, quorum > >>>> arh-ibstorage2-ib,arh-ibstorage3-ib,arh-ibstorage4-ib; noout > >>>> flag(s) set; 47 pgs not deep-scrubbed in time; 14 pgs not scrubbed > >>>> in time > >>>> > >>>> > >>>> This happens on a daily basis several times a day. > >>>> > >>>> Could you please let me know how to fix this annoying problem? > >>>> > >>>> I am running ceph version 15.2.4 > >>>> (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) on > >>>> Ubuntu 18.04 LTS with latest updates. > >>>> > >>>> Thanks > >>>> > >>>> Andrei > >>>> _______________________________________________ > >>>> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send > >>>> an email to ceph-users-leave(a)ceph.io > >>> > >>> > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

I remember exactly this discussion some time ago, where one of the developers gave some more subtle reasons for not using even numbers. The maths sounds simple, with 4 mons you can tolerate the loss of 1, just like with 3 mons. The added benefit seems to be the extra copy of a mon. However, the reality is not that simple. There is apparently some kind of subtlety that has more to do with the physical set-up that makes 4 mons worse than 3 (more likely to lead to loss of service). I do not remember the thread, but it was within the last year. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Janne Johansson <icepic.dz(a)gmail.com> Sent: 29 October 2020 22:07:45 To: Tony Liu Cc: Marc Roos; ceph-users Subject: [ceph-users] Re: frequent Monitor down Den tors 29 okt. 2020 kl 20:16 skrev Tony Liu <tonyliu0592(a)hotmail.com>om>: Well, you can see it the other way around, with 3 configured mons, and only 2 up, you know you have a majority and can go on with writes. With 4 configured mons and only 2 up, it stops because you get the split brain scenario. For a 2DC setup with 2 mons at each place, a split is still fatal. -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io