Elastic Adventures (Part 1)

Jeffery Tay
4 min readMar 29, 2022

Over the course of my work, I had the privilege to work with folks from Govtech’s StackOps team. ElasticStack is certainly not new but it has certainly improved over the years.

This series is to document down my trials with Elastic, starting from design considerations all thru to production use.

[Update: 8 May 2022: Code tested for both Ubuntu 20 and Ubuntu 22, refer to Github link for actual commands as medium replaces some characters]

Designing Elastic for redundancy

Starting off with basic old school, the task is to spin up a 2+1 Node elastic cluster (2 data, 1 master only) so that there is full redundancy + remove the possibility of split brain. All 3 nodes are separately deployed on 3 different physical hardware which are configured with LACP to 2 cross-connect switches for full network redundancy.

N+1 Elastic design + 1 node kibana

Instructions (Elastic 8.1)

Install guide mainly refers to instructions found over at https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

  1. Run the following commands (line by line) to install Elastic on the 3 hosts

wget -qO — https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg — dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

apt-get install apt-transport-https

echo “deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main” | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

apt-get update && sudo apt-get install elasticsearch

Source: https://raw.githubusercontent.com/kwanann/Code-Samples/main/Elastic/Install-On-Ubuntu-20-or-22.txt

2. Wait for the install to be completed and you will be presented with something like the image below. Note down the passwords displayed for each of the nodes.

Elastic Install Completed

3. Update elasticsearch.yml

Now run the following command to modify some parameters in elasticsearch

vi /etc/elasticsearch/elasticsearch.yml

3.1 Name your cluster (in this case log-prod)

3.2 Set networking parameters (allow all network interfaces)

3.3 Set master node configuration for the master node

4. Start ElasticSearch on elastic-data1

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

5. On the elastic-data1, run the following command to generate the enrollment token for the other elastic nodes to join

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node

6. Take note of the token generated and replace [token] in the command below with the actual token. Run this command on elastic-data2 and elastic-master

/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node — enrollment-token [token]

7. On both elastic-data2 and elastic-master, you will receive the following prompt, enter y and press ENTER

8. Now start ElasticSearch on elastic-data2 and elastic-

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

9. Confirm the cluster is up by accessing, login with the password provided in step 2. You will want the status to be green, with 3 nodes and 2 data nodes

Connecting Kibana to Elastic Cluster

  1. On elastic-data1, run the following command to generate a token for Kibana

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

2. Modify the kibana config via

vi /etc/kibana/kibana.yml

3. Enable Kibana to run on all network interface

4. Set Kibana to auto start

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kibana.service
sudo systemctl start kibana.service

5. Access Kibana web interface via

6. Follow prompts on screen to setup Kibana

6.1 if prompted for verification code, run the following command


7. Login with the elastic user and password from step 2

8. When you see this screen, it means your Elastic+Kibana setup is now complete

Next Step: Configuring Fleet and using Elastic Agent to monitor the health of the nodes



Jeffery Tay

Education is in my blood, partnership and coaching is my passion. ¬ L’essentiel est invisible pour les yeux