Logstash provides a powerful mechanism for listening to various input sources, filtering and extracting the fields, and then sending events to a persistence store like ElasticSearch.
Installing Logstash on Ubuntu is well documented, so in this article I will focus on Ubuntu specific steps required for Logstash 2.x and 5.x.
Logstash 2.x Prerequisites
Logstash 2.x requires Java 7 or later, which is easy to satisfy because OpenJDK7 is in the standard repository.
$ sudo apt-get install openjdk-7-jre -y $ sudo update-alternatives --config java $ sudo update-alternatives --config javac $ /usr/bin/java -version
Then add the GPG keys and Logstash 2.x repository location:
$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - $ sudo apt-get install apt-transport-https ca-certificates -y $ echo "deb https://packages.elastic.co/logstash/2.4/debian stable main"| sudo tee -a /etc/apt/sources.list.d/elastic-2.x.list
Logstash 5.x Prerequisites
Logstash 5.x requires Java8 or later, which requires either installing the official Oracle distribution or I prefer OpenJDK. OpenJDK8 is not available in the main Ubuntu repositories, so you must use a PPA like openjdk-r:
$ sudo add-apt-repository ppa:openjdk-r/ppa -y $ sudo apt-get update $ sudo apt-cache policy openjdk-8-jdk $ sudo apt-get install openjdk-8-jdk -y $ sudo update-alternatives --config java $ sudo update-alternatives --config javac $ /usr/bin/java -version
Then add the GPG keys and Logstash 2.x repository location:
$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - $ sudo apt-get install apt-transport-https ca-certificates -y $ echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main"| sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
Logstash Installation
Now that the proper JVM and repository are setup, we can check the version of Logstash that is available for installation (depends on which repository we configured above).
$ sudo apt-get update $ sudo apt-cache policy logstash
If during the apt-get update, there is an error stating ‘The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY …’, then you need to add the ElasticSearch signing key to the trusted keys:
$ sudo apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv D88E42B4
Now run the install and check the status of the service (which should be stopped, not running)
$ sudo apt-get install logstash -y $ sudo service logstash status
For Logstash 2.x, the default home directory is ‘/opt/logstash’ so the version can be checked like:
$ /opt/logstash/bin/logstash --version
For Logstash 5.x, the default home directory is ‘/usr/share/logstash’ so the version can be checked like:
$ /usr/share/logstash/bin/logstash --version
Validation of Logstash
As a quick validation of Logstash functionality, we will invoke it interactively with a very simple conf file. Create the file ‘/usr/share/logstash/simple.conf’ for 5.x or ‘/opt/logstash/simple.conf’ for 2.x with the content below:
input {
stdin { }
}
output {
stdout { codec => rubydebug }
}
Then change directory into the Logstash home directory and start Logstash using the simple configuration file, and it will start listening on standard input for log events, and will send the parsed event to standard output.
$ bin/logstash -f simple.conf
After you see “Pipeline main started”, then type ‘hello world!’ at the console, you should output that looks similar to below:
{
"message" => "hello world!",
"@version" => "1",
"@timestamp" => "2017-05-01T14:10:11.820Z",
"host" => "trusty1"
}
Control-C will break out of the application.
NOTE: If it takes 60+ second before you see “Pipeline main started”, then most likely the issue is that there is not enough entropy in the system for the JVM secure random source. This can be improved by running rng-tools, or modifying ‘$JAVA_HOME/jre/lib/security/java.security’ so that is uses ‘file:/dev/urandom’ instead of ‘file:/dev/random’ for its secure random source.
Logstash as a Service
In order to use Logstash as a service, put your conf files into ‘/etc/logstash/conf.d’. The logs will be output at ‘/var/log/logstash’.
For Logstash 2.x, if you need to make adjustments to heap sizes and other options, modify ‘/etc/default/logstash’.
For Logstash 5.x, if you need to make adjustments there is more control with the following files located in ‘/etc/logstash’: jvm.options, log4j2.properties, logstash.yml, and startup.options.
REFERENCES
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
https://www.elastic.co/guide/en/logstash/2.4/installing-logstash.html
https://askubuntu.com/questions/464755/how-to-install-openjdk-8-on-14-04-lts
http://ubuntuhandbook.org/index.php/2015/01/install-openjdk-8-ubuntu-14-04-12-04-lts/
https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html