This is the fifth in a multi-part series where I explore the process of transforming an existing Graylog install into a resilient and scalable multi-site installation. Start here for Part 1.

At this point we have the dedicated ElasticSearch nodes up and running with the legacy node shut down. MongoDB has been split out and is in a Master/Slave replication mode running on the new nodes. Graylog2 Server is up and running on the new instances with collection inputs created. Graylog2 Web is running on all Graylog2 nodes. By now the only thing that should be left running on the legacy node is the primary Graylog2 Web instance and the log listener on the legacy Graylog2 server.

The final architectural piece to all this madness is to down the legacy Graylog2 instance and re-IP its replacement. Since all of the syslog devices ship their logs to the legacy server this will let us continue to receive logs while we rebalance all the end points.

Our primary goal here is to make sure all of the configs are up to date before we re-IP the server and then restart all the services.

Updating Graylog2 Server

On east-gray01.east.example.com edit /etc/graylog2.conf and confirm the settings:

is_master = false
elasticsearch_node_name = east-gray01
elasticsearch_discovery_zen_ping_unicast_hosts = east-es01.east.example.com:9300,west-es01.west.example.com:9300
rest_listen_uri = http://${IP_ADDRESS}:12900/

Since west-gray01 is replacing the legacy server we'll make it the master Graylog2 instance

is_master = true
elasticsearch_node_name = west-gray01
elasticsearch_discovery_zen_ping_unicast_hosts = east-es01.east.example.com:9300,west-es01.west.example.com:9300
rest_listen_uri = http://${IP_ADDRESS}:12900/

On both east-gray01 and west-gray01 edit /etc/graylog2/web/graylog2-web-interface.conf and make sure it points at the two new Graylog2 servers.

graylog2-server.uris="http://east-es01.east.example.com:12900/,http://west-es01.west.example.com:12900/"

Updating ElasticSearch

The ElasticSearch nodes should be mostly configured correctly already. For the most part we just want to make sure the legacy node is removed. On both east-es01 and west-es01 make sure /etc/elasticsearch/elasticsearch.yml contains

node.master: true
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["east-es01.east.example.com","west-es01.west.example.com"]