Understanding monitor requirements
Hi experts, question about monitors and latency. I am setting up a new cluster and I’d like to have more than one monitor. Unfortunately, the primary site only has two chassis, so to get the third mon, I’ve been trying to bring it up remotely. So far, it’s not working and I wonder if someone knows how I can fit the proverbial square peg in this round hole. This gist[1] has the logs that I have from the third (remote) monitor and a diagram of the network. The logs keep repeating the same errors about slow ops from the first log entry. For comparison, I’ve also included a performance test of etcd that is running on the same three nodes. The secondary site node is slower by an order of magnitude. As for the use case, I mostly expect the primary hardware to be stable and the third mon provides uptime when I need to upgrade either primary node. There will never be any OSD traffic over the site-to-site link. The storage profiles are primarily RBD with static sizes and limited changes. My understanding is that the monitors are only keeping track of PG locations for pools. If that’s the case and the storage is primarily fixed size RBDs, the amount of inter-monitor communication should also be limited since changes are few. Am I mistaken here? Thanks for any input that can be provided! Brian [1] https://gist.github.com/briantopping/3638d665bc60c19e6f3f2005eb555a9d
On Apr 9, 2020, at 8:39 PM, Brian Topping <brian.topping@gmail.com> wrote:
Hi experts, question about monitors and latency.
I think I jumped to a hasty conclusion in my previous email. It’s not about the latency, but about something to do with a problem when mon.c calls an election after synchronizing. In comparing the mon.b startup logs and mon.c during the election phase, I see mon.b go into state `peon` where mon.c goes back to `probing`. See https://gist.github.com/briantopping/3638d665bc60c19e6f3f2005eb555a9d#file-l.... This must have something to do with it, what’s the best solution here? Thanks, Brian
participants (1)
-
Brian Topping