Elasticsearch is an open-source search engine based on Lucene, developed in Java. It provides a distributed and multitenant full-text search engine with an HTTP Dashboard web interface (Kibana). The data is queried, retrieved, and stored in JSON. Elasticsearch is a scalable search engine that can search for all kinds of text documents, including log files.
Fluentd is cross-platform data collection software written in Ruby. It’s an open-source data collector tool that allows you to analyze event logs, application logs, system logs etc.
Kibana is a data visualization interface for Elasticsearch. Kibana provides a pretty dashboard (web interfaces), it allows you to manage and visualize all data from Elasticsearch on your own. It’s not just beautiful but also powerful.
This tutorial will show you step-by-step building the centralized logs using the EFK Stack (Elasticsearch, Fluentd, and Kibana). We will install EFK Stack on the Ubuntu 18.04 system and then try to collect logs from the Ubuntu and CentOS clients to the EFK server.
Prerequisites
- 3 Servers
- efk-master 10.0.15.10 Ubuntu 18.04
- client01 10.0.15.11 Ubuntu 18.04
- client02 10.0.15.12 CentOS 7.5
- Root privileges
What will we do?
- Global Server Configuration
- Set up NTP
- Increase Max File Descriptor
- Optimize Network Kernel Parameters
- EFK Server Setup
- Install Java
- Install and Configure Elasticsearch
- Install and Configure Kibana
- Install and Configure Nginx as Reverse-Proxy for Kibana
- Install and Configure Fluentd
- Setup Ubuntu and CentOS Clients
- Install and Configure Fluentd
- Configure Rsyslog
- Testing
Step 1 – Global Server Configuration
In this step, we will prepare all servers Ubuntu and CentOS for Fluentd installation. So run all of the commands below on all 3 servers.
Set up NTP
For this guide, we will use ntpd to set up the NTP server.
Install NTP packages using commands below.
On Ubuntu servers.
sudo apt install ntp ntpdate -y
On CentOS server.
sudo yum install ntp ntpdate -y
And after the installation is complete, edit the NTP configuration file ‘/etc/ntp.conf’ using vim editor.
vim /etc/ntp.conf
Now choose your continent area where the server is located by visiting the NTP pool list. Comment the default pool and change with your own pool as below.
server 0.id.pool.ntp.org iburst server 1.id.pool.ntp.org iburst server 2.id.pool.ntp.org iburst server 3.id.pool.ntp.org iburst
Save and exit.
Now restart the ntpd services.
On Ubuntu servers.
systemctl restart ntp
On CentOS server.
systemctl restart ntpd
The NTP server configuration has been completed.
Increase Max File Descriptor
The default max file descriptor on the Linux server is ‘1024’. And for the fluentd installation, setting up the file descriptor to ‘65536’ is recommended.
Go to the ‘/etc/security’ directory and edit the configuration file ‘limits.conf’.
cd /etc/security/ vim limits.conf
Paste configuration below to the end of the line.
root soft nofile 65536 root hard nofile 65536 * soft nofile 65536 * hard nofile 65536
Save and exit.
Optimize Network Kernel Parameters
Edit the ‘/etc/sysctl.conf’ file using vim.
vim /etc/sysctl.conf
And paste configuration below.
net.core.somaxconn = 1024 net.core.netdev_max_backlog = 5000 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_wmem = 4096 12582912 16777216 net.ipv4.tcp_rmem = 4096 12582912 16777216 net.ipv4.tcp_max_syn_backlog = 8096 net.ipv4.tcp_slow_start_after_idle = 0 net.ipv4.tcp_tw_reuse = 1 net.ipv4.ip_local_port_range = 10240 65535
Save and exit.
Note:
- These kernel options were originally taken from the presentation “How Netflix Tunes EC2 Instances for Performance†by Brendan Gregg, Senior Performance Architect at AWS re:Invent 2017.
Now reload the kernel parameters using sysctl command.
sysctl -p
The global server configuration for the FLuentd installation has been completed.
Step 2 – EFK Stack Server Setup
In this step, we will install and configure the EFK Stack on the ‘efk-master’ server. This step will cover the java, elasticsearch, kibana, and fluentd installation on an Ubuntu system.
Install Java
We will install java from the PPA webupd8team repository.
Install the ‘software-properties-common’ package and add the java repository.
sudo apt install software-properties-common apt-transport-https -y sudo add-apt-repository ppa:webupd8team/java -y
Now install the java8-installer.
sudo apt install oracle-java8-installer -y
When the installation is complete, check the java version.
java -version
Java 1.8 installed on the system.
Next, we will configure the java environment. Check the java binary file using the command below.
update-alternatives --config java
And you will get the java binary file on the ‘/usr/lib/jvm/java-8-oracle’ directory.
Now create the profile file ‘java.sh’ under the ‘profile.d’ directory.
vim /etc/profile.d/java.sh
Paste java environment configuration below.
#Set JAVA_HOME JAVA_HOME="/usr/lib/jvm/java-8-oracle" export JAVA_HOME PATH=$PATH:$JAVA_HOME export PATH
Save and exit.
Make the file executable and load the configuration file.
chmod +x /etc/profile.d/java.sh source /etc/profile.d/java.sh
Now check the java environment using the command below.
echo $JAVA_HOME
And you will get the java directory is located at ‘/usr/lib/jvm/java-8-oracle’ directory.
Install Elasticsearch
After installing Java, we will install the first component of the EFK Stack (we will install the elasticsearch).
Add the elastic key and repository to the system.
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
Now update the repository and install the elasticsearch package using the command below.
sudo apt update sudo apt install elasticsearch -y
After the installation is complete, go to the ‘/etc/elasticsearc’ directory and edit the configuration file ‘elasticsearch.yml’.
cd /etc/elasticsearch/ vim elasticsearch.yml
Uncomment the ‘network.host’ line and change the value to ‘localhost’, and uncomment the ‘http.port’ a line for the elasticsearch port configuration.
network.host: localhost http.port: 9200
Save and exit.
Now start the elasticsearch service and enable the service to launch everytime at system boot.
systemctl start elasticsearch systemctl enable elasticsearch
Elasticsearch is now up and running, check it using the netstat and curl commands below.
netstat -plntu curl -XGET 'localhost:9200/?pretty'
Now you will get the elasticsearch version ‘6.2.4’ is running on the default port ‘9200’.
Install and Configure Kibana
The second component is a Kibana Dashboard. We will install the Kibana dashboard from the elastic repository, and configure the kibana service to run on the localhost address.
Install Kibana dashboard using the apt command below.
sudo apt install kibana -y
Now go to the ‘/etc/kibana’ directory and edit the configuration file ‘kibana.yml’.
cd /etc/kibana/ vim kibana.yml
Uncomment the lines ‘server.port’, ‘server.host’, and ‘elasticsearch.url’.
server.port: 5601 server.host: "localhost" elasticsearch.url: "http://localhost:9200"
Save and exit.
Now start the kibana service and enable it to launch everytime at system boot.
sudo systemctl enable kibana sudo systemctl start kibana
The kibana dashboard is now up and running on the ‘localhost’ address and the default port ‘5601’. Check it using netstat command below.
netstat -plntu
Kibana installation has been completed.
Install and Configure Nginx as Reverse-Proxy for Kibana
In this tutorial, we will be using the Nginx web server as a reverse proxy for the Kibana Dashboard.
Install Nginx and the ‘apache2-utils’ packages to the system.
sudo apt install nginx apache2-utils -y
After the installation is complete, go to the ‘/etc/nginx’ configuration directory and create new virtual host file named ‘kibana’.
cd /etc/nginx/ vim sites-available/kibana
Paste the following Nginx virtual host configuration there.
server {
listen 80;
server_name efk-stack.io;
auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/.kibana-user;
location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
Save and exit.
Next, we will create new basic authentication web server for accessing the Kibana dashboard. We will create the basic authentication using the htpasswd command as shown below.
sudo htpasswd -c /etc/nginx/.kibana-user elastic
TYPE THE ELASTIC USER PASSWORD
Activate the kibana virtual host and test all nginx configuration.
ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/ nginx -t
Make sure there is no error, now start the Nginx service and enable it to launch everytime at system boot.
systemctl enable nginx systemctl restart nginx
Nginx installation and configuration as a Reverse-proxy for the Kibana dashboard has been completed.
Install and Configure Fluentd
Now we will install Fluentd packages using the ‘Debian stretch 9’ packages. We will install the fluentd packages from the repository, and then configure fluentd for secure forwarding data over SSL.
Download and install fluentd using the Debian installer script as shown below.
curl -L https://toolbelt.treasuredata.com/sh/install-debian-stretch-td-agent3.sh | sh
And after the installation is complete, we need to add new fluentd plugins elasticsearch and secure-forward.
Install fluentd plugins elasticsearch and secure_forward using commands below.
sudo /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch --no-document sudo /usr/sbin/td-agent-gem install fluent-plugin-secure-forward --no-document
Fluentd and fluentd plugins have been installed.
Next, we need to generate new certificate file for the secure logs transfer from clients to the efk-master server.
Generate the certificate file using the command below.
cd /opt/td-agent/ ./embedded/lib/ruby/gems/2.4.0/bin/secure-forward-ca-generate /etc/td-agent/ hakase321
The certificate files ‘ca_cert.pem’ and ‘ca_key.pem’ with password ‘hakase321’ has been generated to the ‘/etc/td-agent’ directory.
ls -lah /etc/td-agent/
Now go to the ‘/etc/td-agent’ directory, backup the original configuration file ‘td-agent.conf’, and create the new one.
cd /etc/td-agent/ mv td-agent.conf td-agent.conf.orig
vim td-agent.conf
Paste the following configuration there.
<source> @type secure_forward shared_key FLUENTD_SECRET self_hostname efk-master secure yes cert_auto_generate yes ca_cert_path /etc/td-agent/ca_cert.pem ca_private_key_path /etc/td-agent/ca_key.pem ca_private_key_passphrase hakase321 </source> <match *.**> @type elasticsearch logstash_format true logstash_prefix fluentd <buffer> flush_interval 10s </buffer> </match>
Save and exit.
Test the fluentd configuration and make sure there is no error, then restart the service.
td-agent --dry-run systemctl restart td-agent
Fluentd is now up and running on the Ubuntu system, check it using the netstat command below.
netstat -plntu
And you will get the default port ‘24284’ is on the ‘LISTEN’ state – it’s used by the ‘secure_forward’ source.
The EFK Stack server setup has been completed.
Step 3 – Set up Ubuntu and CentOS Clients
In this step, we will configure the Ubuntu 18.04 and CentOS 7 clients. We will install and configure Fluentd agent on both servers to collects server logs, then send all logs to the ‘efk-master’ server through the ‘secure_forward’ SSL.
Configure Hosts File
Edit the ‘/etc/hosts’ file on both systems and add the efk-master server IP address.
vim /etc/hosts
Paste configuration below.
10.0.15.10 efk-master efk-master
Save and exit.
Install and Configure Fluentd
Now install Fluentd using the installer script as shown below.
For Ubuntu 18.04 system.
curl -L https://toolbelt.treasuredata.com/sh/install-debian-stretch-td-agent3.sh | sh
For CentOS 7 system.
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
After the installation is complete, install the ‘secure_forward’ plugin using ‘td-agent-gem’ command below.
sudo /usr/sbin/td-agent-gem install fluent-plugin-secure-forward --no-document
Fluentd packages with the ‘secure_forward’ plugins have been installed.
Now we need to download the ‘ca_cert.pem’ certificate file from the ‘efk-master’ server to all clients.
Download the ‘ca_cert.pem’ certificate using scp.
scp root@efk-master:/etc/td-agent/ca_cert.pem /etc/td-agent/ TYPE THE ROOT PASSWORD
The ‘ca_cert.pem’ certificate file has been downloaded to the ‘/etc/td-agent/’ directory.
ls -lah /etc/td-agent/
Next, we need to create new ‘td-agent.conf’ configuration file for the client. Go to the ‘/etc/td-agent’ directory, back up the original file and create a new one.
cd /etc/td-agent/ mv td-agent.conf td-agent.conf.orig
vim td-agent.conf
Paste the following configuration there.
<source>
@type syslog
port 42185
tag client01
</source>
<match *.**>
@type secure_forward
shared_key FLUENTD_SECRET
self_hostname "client01"
secure yes
ca_cert_path /etc/td-agent/ca_cert.pem
<server>
host efk-master
port 24284
</server>
</match>
Save and exit.
Note:
- Change the ‘self_hostname’ value with your clients’ hostname.
Now test the Fluentd configuration and make sure there is no error, then restart the fluentd service.
td-agent --dry-run systemctl restart td-agent
The fluentd service is now up and running on client Ubuntu and CentOS servers. Check it using netstat command below.
netstat -plntu
And you will get the port ‘42185’ is on LISTEN state used by the fluentd service.
Configure Rsyslog
Edit the rsyslog configuration file ‘/etc/rsyslog.conf’ using vim editor.
vim /etc/rsyslog.conf
Paste the following configuration to the end of the line.
*.* @127.0.0.1:42185
Save and exit, then restart the rsyslog service.
systemctl restart rsyslog
The client servers configuration Ubuntu and CentOS have been completed.
Step 4 – Testing
Open your web browser and type the EFK Stack URL http://efk-stack.io.
Now you will be asked for the user and password for basic auth login from Nginx web server, type the ‘elastic’ user with your password.
And you will get the Kibana Dashboard.
Click the ‘Set up index patterns’ button, then define the index pattern to ‘fluentd-*’.
Click the ‘Next step’ button.
For the index pattern configure settings, choose the filter field name for ‘@timestamp’.
Click the ‘Create index pattern’ button.
And the fluentd index pattern has been created.
Click the ‘Discover’ menu on the left to get all server logs.
Below is an example for failed ssh login on both clients Ubuntu and CentOS.
‘client01’ Ubuntu 18.04 ssh failed password log.
‘client02’ CentOS 7 ssh failed password log.
Installation and configuration for the Centralized Logs using EFK Stack (Elasticsearch, Fluentd, and Kibana) on Ubuntu 18.04 have been completed successfully.