ELK Stack Installation on Ubuntu 14.04

% Log Management Part 1: Using ELK Stack
%
% Network Monitoring & Management

# Notes

* Commands preceded with "$" imply that you should execute the command as
  a general user - not as root.
* Commands preceded with "#" imply that you should be working as root.

# Exercise

###Installation###

##Edit sources.list file,comment out the listed mirrors and add the lines below:

$ sudo vi /etc/apt/sources.list
#------------------------------------------------------------------------------#
#                            OFFICIAL UBUNTU REPOS                             #
#------------------------------------------------------------------------------#


###### Ubuntu Main Repos
deb http://ke.archive.ubuntu.com/ubuntu/ trusty main 
deb-src http://ke.archive.ubuntu.com/ubuntu/ trusty main 

###### Ubuntu Update Repos
deb http://ke.archive.ubuntu.com/ubuntu/ trusty-security main 
deb http://ke.archive.ubuntu.com/ubuntu/ trusty-updates main 
deb-src http://ke.archive.ubuntu.com/ubuntu/ trusty-security main 
deb-src http://ke.archive.ubuntu.com/ubuntu/ trusty-updates main 


$ sudo apt-get update && apt-get dist-upgrade
$ sudo apt-get autoremove
$ apt-get install software-properties-common

##Install Java##
$ sudo add-apt-repository -y ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get -y install oracle-java8-installer
$ sudo java -version

##Install Elasticsearch##

##Run the following command to import the Elasticsearch public GPG key into apt:
$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

##Navigate to your home directory:
cd /home/hostX

##Download Elasticsearch .deb file:
$ wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.4.0/elasticsearch-2.4.0.deb

##Update your apt package database:
$ sudo apt-get update

##Install Elasticsearch with this command:
$ dpkg -i elasticsearch-2.4.0.deb

##Elasticsearch is now installed. Let's edit the configuration:

$ cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.orig

$ sudo vi /etc/elasticsearch/elasticsearch.yml

##Restrict outside access to ES on port 9200(HTTP);bind to localhost
network.host: localhost

##Save and exit elasticsearch.yml.

##Now start Elasticsearch:
$ sudo service elasticsearch start

##Start Elasticsearch on boot up:
$ sudo update-rc.d elasticsearch defaults 95 10

##Install Kibana##

##Create the Kibana source list:
$ echo "deb http://packages.elastic.co/kibana/4.4/debian stable main" | sudo tee -a /etc/apt/sources.list.d/kibana-4.4.x.list

##Update your apt package database:
$ sudo apt-get update

##Install Kibana with this command:
$ sudo apt-get -y install kibana

##Configure Kibana:
$ sudo cp /opt/kibana/config/kibana.yml /opt/kibana/config/kibana.yml.orig 
$ sudo vi /opt/kibana/config/kibana.yml

##Find server.host, and replace the IP address ("0.0.0.0" by default) with "localhost":
server.host: "localhost"

##Enable the Kibana service, and start it:

$ sudo update-rc.d kibana defaults 96 9
$ sudo service kibana start
$ sudo service kibana status

##Set up Reverse Proxy##

##Install Nginx##
$ sudo apt-get install nginx apache2-utils

##Create an admin user, called "tradmin", that can access the Kibana web interface:

$ sudo htpasswd -c /etc/nginx/htpasswd.users tradmin
Enter a password at the prompt (use "k3yb@na")

##Open the Nginx default server block in your favorite editor:
$ sudo vi /etc/hosts
#add
10.10.0.30	host30.ws.nsrc.org host30

$ cp /etc/nginx/sites-available/default /etc/nginx/sites-available/default.orig

##Delete the file's contents, and paste the following code block into the file. Be sure to update the server_name to match your server's name:

$ sudo vi /etc/nginx/sites-available/default
    server {
        listen 80;

        server_name example.com;

        auth_basic "Restricted Access";
        auth_basic_user_file /etc/nginx/htpasswd.users;

        location / {
            proxy_pass http://localhost:5601;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;        
        }
    }

##Save and exit
//This configures Nginx to direct your server's HTTP traffic to the Kibana application, which is listening on localhost:5601. Also, Nginx will use the htpasswd.users file, created earlier, and require basic authentication.

##Restart Nginx
$ sudo service nginx status
$ sudo service nginx restart

##Access Kibana Web Interface
http://ip_address/


##Install Logstash##

##Create the source list:
$ echo 'deb http://packages.elastic.co/logstash/2.2/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash-2.2.x.list

##Update your apt package database:
$ sudo apt-get update

##Install Logstash:
$ sudo apt-get install logstash

##Configure Logstash##
//Config files are in JSON format in three sections;inputs, filters and outputs


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...sample logstash configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
##Inputs:
udp {
    port => "514"
    type => "Router"
  }


##Filters:
##create a configuration file where we will add a filter for syslog messages:
$ sudo vi /etc/logstash/conf.d/10-syslog-filter.conf

    filter {
      if [type] == "syslog" {
        grok {
          match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
          add_field => [ "received_at", "%{@timestamp}" ]
          add_field => [ "received_from", "%{host}" ]
        }
        syslog_pri { }
        date {
          match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
      }
    }


##Outputs:
output {
  elasticsearch { hosts => localhost }
  stdout { codec => rubydebug }
}   


//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~