This is the fourth in a multi-part series where I explore the process of transforming an existing Graylog install into a resilient and scalable multi-site installation. Start here for Part 1.

The MongoDB section actually gave me the biggest set of problems. Part of this is because of my utter lack of familiarity with Mongo and partially because of Mongo's design. Originally I hoped to go for a full Master/Master fully replicated environment. This would meet the fully redundant design that I was aiming for but during configuration I ran into a bit of snag. By design MongoDB requires a minimum of 3 nodes so as not to result in hung juries during master elections. This means that for my fully redundant geographically diverse setup I would need a minimum of 6 Mongo nodes, 3 for east coast and 3 for west coast.

Personally, I find the idea of running 6 servers to store indices, metadata, and configuration data a bit much. Particularly considering the rest of the environment is planned to be a total of 4-6 systems. Due to this we opted to scale back a bit and go for Master/Slave replication. This still gives us site diversity, and a backup node, but requires manual effort to promote the slave to master.

The process of promoting a single MongoDB instance to a master and adding a slave is actually very straightforward. The basic workflow here is:

Convert the legacy standalone to a master node
Add a new mongod instance as a slave
Wait for replication to occur
Shutdown the mongod instance on legacy standalone
Promote new slave to master
Add the other new mongod instance as slave
Wait for replication to occur
Reconfigure graylog2-server and graylog2-web-server instances to use new Mongo nodes.

Convert to Master/Slave and Replicate

In order to replicate application data across we have to set up our legacy system to listen on a non-localhost address, and to listen for slave connections. The configs are actually pretty easy.

The biggest caveat to keep in mind is that the Graylog2 Server standalone instance will be configured to hit mongodb on localhost. During the replication and promotion Graylog2 will not be able to access the database instance.

First edit /etc/mongodb.conf on the legacy master

Legacy Master

dbpath=/var/lib/mongodb
logpath=/var/log/mongodb/mongodb.log
logappend=true
bind_ip = ${public_ip}
fork = true
master = true

Then edit /etc/mongodb.conf on the new host

New Slave

dbpath=/var/lib/mongodb
logpath=/var/log/mongodb/mongodb.log
logappend=true
bind_ip = ${public_ip}
journal=true
slave = true
source = ${master_node_address}

Replication Times

Since very little information is actually stored in Mongo replication should happen fairly fast. We can monitor this on the filesystem by keeping track of the files stored in /var/lib/mongodb. Each database will, almost definitely, consume multiple files of the form graylog2.#. Ignore the files starting with local, those are instance specific. Once the files settle down the database will have finished replicating and we can move on.

Slave Promotion

When it comes time to actually fiddle with the master node assignment we have several options to choose from. Since my plan is to add yet another mongodb instance I opted to go with the Failing over to a slave (Promotion). In a larger environment performing an inversion would allow us to bring all the nodes into the replication cluster before removing the legacy system, but in my use case it was just as easy to kill off the legacy box at the beginning.

Stop the mongodb instance on both master and slave: initctl stop mongodb
Edit the /etc/mongodb.conf file on the new node setting it as the master

dbpath=/var/lib/mongodb
logpath=/var/log/mongodb/mongodb.log
logappend=true
bind_ip = ${public_ip}
fork = true
master = true

Backup the local.# files: mkdir /var/lib/mongodb_local_bak && mv /var/lib/mongodb/local* /var/lib/mongodb_local_bak/.
Start up the mongodb instance on the new master: initctl stop mongodb
Restart the Graylog2 Server instances to connect to the new mongo instance: initctl restart graylog2-server

Add New MongoDB Instance as Slave

Now that the old master is out of the way we can add our new secondary as a slave. This is done just like before

dbpath=/var/lib/mongodb
logpath=/var/log/mongodb/mongodb.log
logappend=true
bind_ip = ${public_ip}
journal=true
slave = true
source = ${new_master_node_address}

Once we've made these changes we can start mongod on the new slave and let replication happen.