% Log Management Part 1: Using rsyslog % % Network Monitoring & Management # Notes * Commands preceded with "$" imply that you should execute the command as a general user - not as root. * Commands preceded with "#" imply that you should be working as root. * Commands with more specific command lines (e.g. "R1X>") imply that you are executing commands on remote equipment, or within another program. # Exercise The routers are able to send syslog messages to multiple destinations, so that 1 router can send messages to 4 or even 5 destinations. We therefore need to configure the router to send messages to each of the PCs in the group. ## Configure sending of syslog messages from your group's router Configure your virtual router to send syslog messages to every server in your group. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ $ telnet router_ip R1X> enable R1X# config terminal R1X(config)# logging host 10.10.X.Y transport udp port 514 ... where X.Y is the IP of your host (group + number, example hostX = 10.10.0.2). R1X(config)# logging facility local0 R1X(config)# logging userinfo R1X(config)# logging severity 5 R1X(config)# exit R1X# write memory R1X# exit ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Now run `show logging` to see the summary of the log configuration. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ R1X# show logging ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The other participants in your group will be doing the same thing, so you should not be surprised if you see other destinations as well in the output of "show logging" - Press SPACE to page through the output Logout from the router (exit): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ R1X# exit ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ That's it. The router should now be sending UDP SYSLOG packets to your PC on port 514. To verify this log in on your host as user sysadm (if you have not already done so) do the following: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ $ sudo -s # apt-get install tcpdump (if already installed dont worry) # tcpdump -s0 -nv -i eth0 udp port 514 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Then have one person in your group log back in on the router and do the following: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ $ telnet router_ip R1X> enable R1X# config terminal R1X(config)# exit R1X> exit ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You should see some output on your host's screen from `tcpdump`. It should look something like: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11:20:24.942289 10.10.1.254.63515 > 10.10.1.1.514: SYSLOG local0.notice, length: 110 11:20:24.944376 10.10.1.254.53407 > 10.10.1.1.514: SYSLOG local0.notice, length: 102 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When you have seen this, hit Ctrl-C to exit tcpdump. Aside: to learn more about tcpdump type "man tcpdump" at the command line Now you can configure the logging software on your PC to receive this information and log it to a new set of files. ## Configure rsyslog Be sure you are logged in to your virtual machine and that you are the root user. Edit the file /etc/rsyslog.conf: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # cp /etc/rsyslog.conf /etc/rsyslog.conf.orig # editor /etc/rsyslog.conf ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ...and find and un-comment the following lines (that is, remove the initial '#' only) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # provides UDP syslog reception #$ModLoad imudp #$UDPServerRun 514 change to: $ModLoad imudp $UDPServerRun 514 # provides TCP syslog reception #$ModLoad imtcp #$InputTCPServerRun 514 change to: $ModLoad imtcp $InputTCPServerRun 10514 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Then change this line: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ $PrivDropToGroup syslog change to: $PrivDropToGroup adm ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Then save the file and exit. Now, create a file named "/etc/rsyslog.d/30-routerlogs.conf" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # editor /etc/rsyslog.d/30-routerlogs.conf ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ... and add the following lines (carefully COPY and PASTE): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ $template RouterLogs,"/var/log/network/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%-%$HOUR%.log" local0.* -?RouterLogs & ~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ PLEASE double check (verify) that what you have pasted is the SAME as what is above. In particular, make sure that you are using TAB and not SPACE between "template" and "RouterLogs", and also between "local0.*" and "-?RouterLogs". If the above is not pasted correctly, it will NOT work. Save and exit, then do: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # mkdir /var/log/network # chown syslog:adm /var/log/network # chmod g+w /var/log/network ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Restart rsyslog: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # service rsyslog restart ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ## Test syslog To be sure there are some logging messages log back in to the router, and run some "config" commands, then logout. e.g. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ $ telnet router_ip R1X> enable R1X# config terminal R1X(config)# exit R1X> exit ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Be sure you log out of the router when you are finished. If too many people log in without logging out then others cannot gain access to the router. On your host, See if messages are starting to appear under `/var/log/network////` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ $ cd /var/log/network $ ls $ cd $ ls ... this will show you the directory for the month ... cd into this directory $ ls ... repeat for the next level (the day of the month) $ ls ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Then use 'tail' to look at the log file(s) in this directory. The names are dynamic based on the sender and the host, so use the file that you see. It may be something like this: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ $ ls R11-16.log $ tail 10.10.0.201-05.log ... logging messages are shown ... Sep 7 05:44:43 10.10.0.201 3209: *Sep 7 05:06:00.200: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by unknown on vty0 (10.10.0.219) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ## Troubleshooting rsyslog If no files are appearing under the /var/log/network directory, then another command to try while logged into the router, in config mode, is to shutdown / no shutdown a Loopback interface, for example: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ $ telnet router_ip R1X> enable R1X# conf t R1X(config)# interface Gi0/2 R1X(config-if)# shutdown ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ wait a few seconds ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ R1X(config-if)# no shutdown ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Then exit, and save the config ("write mem"): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ R1X(config-if)# exit R1X(config)# exit R1X# write memory rtr1# exit ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Check the logs under `/var/log/network` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # cd /var/log/network # ls ...follow the directory trail ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Still no logs? Try the following command to send a test log message locally: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # logger -p local0.info "Hello World\!" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If a file has not been created yet under `/var/log/network`, then check your configuration for typos. Don't forget to restart the rsyslog service each time you change the configuration. What other commands can you think of that you can run on the router (BE CAREFUL!) that will trigger syslog messages? You could try logging in on the router and typing an incorrect password for "enable". Be sure that you do an "ls" command in your logging directory to see if a new log file has been created at some point.