How to Install and Configure ELK Stack on Ubuntu and Debian

ELK stack consists of a set of applications for retrieving and managing log files. In the software development industry, log files play a vital role to identify the problem and troubleshoot the issue. ELK stack is a collection of different open-source application tools such as Elasticsearch, Kibana, and Logstash. ELK can be used to collect, search and visualize the logs generated from any source in any pattern using a query. In this article, we will learn how to install and configure the ELK stack on Ubuntu and Debian.

Prerequisites:

1. Fresh Ubuntu 20.04 or Debian 10 Server
2. Root Privileged Account
3. Proper Internet connection

Install Java

Installation of ELK stack requires a Java environment. Run the following command to install java on Ubuntu/Debian

$ sudo apt install openjdk-8-jdk

Verify the installation by checking the java version

$ java -version

Output:

Install and configure Elasticsearch

Once java is installed, now it's time to install and configure Elasticsearch. Since Elasticsearch packages are not available by default on Ubuntu/Debian we need to add the elasticsearch apt repository. Run the following command to add the GPG repository key.

$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Now create the repository file using the command.

$ echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Once the repository file is created, elasticsearch can be installed using the command.

$ sudo apt update

$ sudo apt install elasticsearch

The default configuration file of elasticsearch is located at /etc/elasticsearch/elasticsearch.yml . Use any text editor and uncomment the lines:

network.host: localhost

http.port: 9200

Start and enable elasticsearch

$ sudo systemctl start elasticsearch

$ sudo systemctl enable elasticsearch

Run the following command to view Elasticsearch status and details

$ curl -X GET "localhost:9200"

Output:

Install and configure Logstash

Logstash package is available by default in Ubuntu/Debian systems. Run the following command to install.

$ sudo apt install logstash

Start and enable the service

$ sudo systemctl start logstash

$ sudo systemctl enable logstash

Check the service using the command

$ systemctl status logstash

Default configuration directory of logstash is /etc/logstash/conf.d/ . Once the installation is completed INPUT, FILTER and OUTPUT pipelines can be configured based on required use cases.

Install and configure Kibana

Kibana is a web based GUI tool used for analyzing and parsing the collected logs. Kibana is available in the default repository of Ubuntu/Debian. Run the following command to install the package.

$ sudo apt install kibana

To configure kibana, go to the default configuration directory and uncomment the following lines

$ sudo vim /etc/kibana/kibana.yml

server.port: 5601
server.host: "localhost"
elasticsearch.hosts: ["http://localhost:9200"]

Start and enable the service

$ sudo systemctl start kibana

$ sudo systemctl enable kibana

Allow the kibana port in the firewall

$ sudo ufw allow 5601/tcp

Now access the Kibana dashboard using the url http://localhost:5601

Install and configure filebeat

Filebeat is used for sending logs to elasticsearch and logstash for parsing. Filebeat is available by default in Ubuntu/Debian repository. Run the following command to install.

$ sudo apt install filebeat -y

To configure the filebeat, go to the default configuration directory and comment out the following.

$ sudo vim /etc/filebeat/filebeat.yml

# output.elasticsearch:
# Array of hosts to connect to .
# hosts: ["localhost:9200"]

Uncomment the following line and save the file

output.logstash:
hosts: [“localhost:5044”]

In the next step, enable filebeat system module

$ sudo filebeat modules enable system

Now run the following command to load the index template

$ sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

Start and enable filebeat service

$ sudo systemctl start filebeat

$ sudo systemctl enable filebeat

Check the status

$ sudo systemctl status filebeat

Conclusion

In this article, I have covered how to install and configure the ELK stack on Debian/Ubuntu in the correct way. Also, we have learned how to use different components such as Kibana, Logstash, and Kibana to analyze and visualize the logs from any source.