Elastic Adventures (Part 1)
Over the course of my work, I had the privilege to work with folks from Govtech’s StackOps team. ElasticStack is certainly not new but it has certainly improved over the years.
This series is to document down my trials with Elastic, starting from design considerations all thru to production use.
[Update: 8 May 2022: Code tested for both Ubuntu 20 and Ubuntu 22, refer to Github link for actual commands as medium replaces some characters]
Designing Elastic for redundancy
Starting off with basic old school, the task is to spin up a 2+1 Node elastic cluster (2 data, 1 master only) so that there is full redundancy + remove the possibility of split brain. All 3 nodes are separately deployed on 3 different physical hardware which are configured with LACP to 2 cross-connect switches for full network redundancy.
Instructions (Elastic 8.1)
Install guide mainly refers to instructions found over at https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html
- Run the following commands (line by line) to install Elastic on the 3 hosts
wget -qO — https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg — dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
apt-get install apt-transport-https
echo “deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main” | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
apt-get update && sudo apt-get install elasticsearch
Source: https://raw.githubusercontent.com/kwanann/Code-Samples/main/Elastic/Install-On-Ubuntu-20-or-22.txt
2. Wait for the install to be completed and you will be presented with something like the image below. Note down the passwords displayed for each of the nodes.
3. Update elasticsearch.yml
Now run the following command to modify some parameters in elasticsearch
vi /etc/elasticsearch/elasticsearch.yml
3.1 Name your cluster (in this case log-prod)
3.2 Set networking parameters (allow all network interfaces)
3.3 Set master node configuration for the master node
4. Start ElasticSearch on elastic-data1
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
5. On the elastic-data1, run the following command to generate the enrollment token for the other elastic nodes to join
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node
6. Take note of the token generated and replace [token] in the command below with the actual token. Run this command on elastic-data2 and elastic-master
/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node — enrollment-token [token]
7. On both elastic-data2 and elastic-master, you will receive the following prompt, enter y and press ENTER
8. Now start ElasticSearch on elastic-data2 and elastic-
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
9. Confirm the cluster is up by accessing https://192.168.3.2:9200/_cluster/health, login with the password provided in step 2. You will want the status to be green, with 3 nodes and 2 data nodes
Connecting Kibana to Elastic Cluster
- On elastic-data1, run the following command to generate a token for Kibana
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
2. Modify the kibana config via
vi /etc/kibana/kibana.yml
3. Enable Kibana to run on all network interface
4. Set Kibana to auto start
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kibana.service
sudo systemctl start kibana.service
5. Access Kibana web interface via http://192.168.3.31:5601/
6. Follow prompts on screen to setup Kibana
6.1 if prompted for verification code, run the following command
/usr/share/kibana/bin/kibana-verification-code
7. Login with the elastic user and password from step 2
8. When you see this screen, it means your Elastic+Kibana setup is now complete
Next Step: Configuring Fleet and using Elastic Agent to monitor the health of the nodes