How to Setup Elasticsearch cluster with Multiple Nodes Google Cloud. Elasticsearch cluster is a group of nodes with same cluster name and additional Elasticsearch servers are joined and communicated within each other.
In this guide you are going to learn how to configure multiple instances on Google Cloud with Elasticsearch and make them as a cluster to communicate with each other.
This setup is tested on Google Cloud with multiple instances connected to each other as a cluster for Elasticsearch. So this setup will work on other cloud service providers like AWS or Azure any others.
Prerequisites
Once you have Elasticsearch installed you can follow the below steps to make the configuration.
You can also setup Nginx based setup in your primary Elasticsearch server or instance.
Make sure you use the Internal IP in the host name in the Elasticsearch configuration.
Configure Firewall
By default Elasticsearch service listens on port 9200 and the nodes in a cluster communicate on port 9300. So you need to allow ports 9200 and 9300 in your network.
If you use UFW you need allow these ports.
sudo ufw allow 9200/tcp sudo ufw allow 9300/tcp
In Google Cloud you need to create a firewall rule to allow these ports.
Go to VPC Network >> Firewall rules and click Create Firewall rules.
In Name enter elasticsearch
In Targets select All instances in the network
In Source filter select IP ranges
In Source IP ranges enter 0.0.0.0/0
In Protocols and ports check TCP and enter 9200, 9300
Create Nodes for Elasticsearch Cluster
For creating additional nodes you need to setup additional servers or instances and install Elasticsearch.
You need to install the exact Elasticsearch version that installed in the primary server.
To check the Elasticsearch version in the main instance, SSH to the instance and execute the following command.
curl -XGET 'http://INTERNAL_IP:9200'
You will get an output which shows you the version installed.
To install the specific version you can follow the below steps after installing and configuring Java.
sudo apt-get -y install apt-transport-https curl wget sudo wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add - sudo echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list sudo apt-get update && apt-get install elasticsearch=7.1.1
Once you have installed the correct Elasticsearch versions in all your instances/servers you can proceed to configure it to communicate with one another.
Modify Elasticsearch Configuration for Cluster
SSH to your main Elasticsearch server and make the following changes.
sudo nano /etc/elasticsearch/elasticsearch.yml
Add the following to your configuration.
cluster.name: CLUSTER_NAME node.name: ${HOSTNAME} node.master: true node.data: true discovery.zen.ping.unicast.hosts: ["INTERNAL_IP_SERVER_1", "INTERNAL_IP_SERVER_2"] discovery.zen.minimum_master_nodes: 2
Change the CLUSTER_NAME with the name you wish.
If you wish to name the nodes you can also enter any unique names.
Include all the internal IP addresses of all other servers in the discovery.zen.ping.unicast.hosts
Once you have updated the configurations restart Elasticsearch.
sudo service elasticsearch restart
Now, SSH to additional clusters and edit the elasticsearch.yml file and make the same updates, and restart the service.
Check Cluster State
If everything is configured correctly your cluster should be up and running.
You can check the status using the following command.
{
"cluster_name" : "cluster_name",
"cluster_uuid" : "tDRxIHICTQewxqvRuN8QvQ",
"version" : 1433,
"state_uuid" : "hcFyhKijQtmkjSxKMe0AfQ",
"master_node" : "Tg4JuAtLSQKnAZYVUxOoLg",
"blocks" : { },
"nodes" : {
"Tg4JuAtLSQKnAZYVUxOoLg" : {
"name" : "main-node",
"ephemeral_id" : "x3HLaF-HSWWtljujhwXlxQ",
"transport_address" : "INTERNAL_IP:9300",
"attributes" : {
"ml.machine_memory" : "15734808576",
"xpack.installed" : "true",
"ml.max_open_jobs" : "20",
"ml.enabled" : "true"
}
},
"s-EoZV4KQxO5YUhqYzv3yA" : {
"name" : "node-1",
"ephemeral_id" : "S3SGzhSiSJmJR-Z65A7e_Q",
"transport_address" : "INTERNAL_IP:9300",
"attributes" : {
"ml.machine_memory" : "3862560768",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
}
}
},
. . .
If you see this kind of output your cluster is working fine.
Configure Dedicated Master Nodes
Before configuring dedicated master nodes, make sure that your cluster have at least 3 master-eligible nodes. This is important to avoid a split-brain situation, which can cause inconsistencies in your data in the event of a network failure.
sudo nano /etc/elasticsearch/elasticsearch.yml
Modify the following lines to match the following.
node.master: true node.data: false
Restart Elasticsearch for the changes to take effect.
Configure Dedicated Data Nodes
To configure a dedicated data node which is not master-eligible, edit the respective node’s Elasticsearch configuration and update the configurations to match the settings below.
node.master: false node.data: true
Restart Elasticsearch for the changes to take effect.
Configure Minimum Master Nodes
It is important to set the minimum number of master-eligible nodes that need to be running for the cluster to function normally, which is sometimes referred to as quorum.
To calculate the number of minimum master nodes your cluster should have, calculate n / 2 + 1, where n is the total number of “master-eligible” nodes in your healthy cluster, then round the result down to the nearest integer.
For example, for a 3-node cluster, the quorum is 2.
Now you have configured your Elasticsearch cluster with some basic optimizations and running healthy.
Conclusion
Now you have learned how to setup Elasticsearch and configure a cluster in a production based setup.
Thanks for your time. If you face any problem or any feedback, please leave a comment below.
