Elastic Adventures (Part 1)

4 min readMar 29, 2022

Over the course of my work, I had the privilege to work with folks from Govtech’s StackOps team. ElasticStack is certainly not new but it has certainly improved over the years.

This series is to document down my trials with Elastic, starting from design considerations all thru to production use.

[Update: 8 May 2022: Code tested for both Ubuntu 20 and Ubuntu 22, refer to Github link for actual commands as medium replaces some characters]

Designing Elastic for redundancy

Starting off with basic old school, the task is to spin up a 2+1 Node elastic cluster (2 data, 1 master only) so that there is full redundancy + remove the possibility of split brain. All 3 nodes are separately deployed on 3 different physical hardware which are configured with LACP to 2 cross-connect switches for full network redundancy.

Instructions (Elastic 8.1)

Install guide mainly refers to instructions found over at https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

Run the following commands (line by line) to install Elastic on the 3 hosts

wget -qO — https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg — dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
apt-get install apt-transport-https
echo “deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main” | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
apt-get update && sudo apt-get install elasticsearch

Source: https://raw.githubusercontent.com/kwanann/Code-Samples/main/Elastic/Install-On-Ubuntu-20-or-22.txt

2. Wait for the install to be completed and you will be presented with something like the image below. Note down the passwords displayed for each of the nodes.

3. Update elasticsearch.yml

Now run the following command to modify some parameters in elasticsearch

vi /etc/elasticsearch/elasticsearch.yml

3.1 Name your cluster (in this case log-prod)

3.2 Set networking parameters (allow all network interfaces)

3.3 Set master node configuration for the master node

4. Start ElasticSearch on elastic-data1

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

5. On the elastic-data1, run the following command to generate the enrollment token for the other elastic nodes to join

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node

6. Take note of the token generated and replace [token] in the command below with the actual token. Run this command on elastic-data2 and elastic-master

/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node — enrollment-token [token]

7. On both elastic-data2 and elastic-master, you will receive the following prompt, enter y and press ENTER

8. Now start ElasticSearch on elastic-data2 and elastic-

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

9. Confirm the cluster is up by accessing https://192.168.3.2:9200/_cluster/health, login with the password provided in step 2. You will want the status to be green, with 3 nodes and 2 data nodes