Logstash provides a powerful mechanism for listening to various input sources, filtering and extracting the fields, and then sending events to a persistence store like ElasticSearch.
Installing Logstash on Ubuntu is well documented, so in this article I will focus on Ubuntu specific steps required for Logstash 6.x on Ubuntu 16.04.
Logstash 6.x Prerequisites
Logstash 6.x requires Java8 or later, which is available from the the main Ubuntu repositories on Ubuntu 16.04 (OpenJDK8 requires ppa on Ubuntu 14.04).
$ sudo apt-get install openjdk-8-jdk
Then add the GPG keys and Logstash 2.x repository location:
$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - $ sudo apt-get install apt-transport-https ca-certificates -y $ echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main"| sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
Logstash Installation
Now that the proper JVM and repository are setup, we can check the version of Logstash that is available for installation (depends on which repository we configured above).
$ sudo apt-get update $ sudo apt-cache policy logstash
If during the apt-get update, there is an error stating ‘The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY …’, then you need to add the ElasticSearch signing key to the trusted keys:
$ sudo apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv D88E42B4
Now run the install and check the status of the service (which should be stopped, not running). This is Systemd, since we are on Ubuntu 16.04.
$ sudo apt-get install logstash -y $ sudo systemctl status logstash
For Logstash 6.x, the default home directory is ‘/usr/share/logstash’ so the version can be checked like:
$ /usr/share/logstash/bin/logstash --version
Validation of Logstash
As a quick validation of Logstash functionality, we will invoke it interactively with a very simple conf file. Create the file ‘/usr/share/logstash/simple.conf’ for 6.x with the content below:
input {
stdin { }
}
output {
stdout { codec => rubydebug }
}
Then change directory into the Logstash home directory and start Logstash using the simple configuration file, and it will start listening on standard input for log events, and will send the parsed event to standard output.
$ cd /usr/share/logstash $ sudo bin/logstash -f simple.conf
After you see “Pipeline main started”, then type ‘hello world!’ at the console, you should output that looks similar to below:
{
"message" => "hello world!",
"@version" => "1",
"@timestamp" => "2018-03-01T14:10:11.820Z",
"host" => "xenial1"
}
Control-C will break out of the application.
NOTE: If it takes 60+ second before you see “Pipeline main started”, then most likely the issue is that there is not enough entropy in the system for the JVM secure random source. This can be improved by installing ‘rng-tools’, or modifying ‘$JAVA_HOME/jre/lib/security/java.security’ so that is uses ‘file:/dev/urandom’ instead of ‘file:/dev/random’ for its secure random source.
Logstash as a Service
In order to use Logstash as a service, put your conf files into ‘/etc/logstash/conf.d’. The logs will be output at ‘/var/log/logstash’.
For Logstash 6.x, if you need to make adjustments there is more control with the following files located in the directory ‘/etc/logstash’: jvm.options, log4j2.properties, logstash.yml, and startup.options.
REFERENCES
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
http://ubuntuhandbook.org/index.php/2015/01/install-openjdk-8-ubuntu-14-04-12-04-lts/
https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html
NOTES
If you are on Ubuntu 14.04 you can install OpenJDK 8 from the ppa
$ sudo add-apt-repository ppa:openjdk-r/ppa -y $ sudo apt-get update $ sudo apt-cache policy openjdk-8-jdk $ sudo apt-get install openjdk-8-jdk -y $ sudo update-alternatives --config java $ sudo update-alternatives --config javac $ /usr/bin/java -version