Friday, June 6, 2008

Some snort login kung-fu...

I was recently playing around with my .bash_profile file looking for new ways to alert myself as well as my team to problems with production snorts. I ended up with two little tricks that I have found really useful and I figured I would share.

For those that don't know, the .bash_profile file is an sh script that runs at user login. At a bare minimum it sets the users PATH, but it can be used for a whole lot more. It's located in the root of the users home directory. Ex: /home/snort/.bash_profile, or /root/.bash_profile

Before I go any further I will tell you that both of these tricks are obviously reactive in nature. They only let you know there is a problem the next time you log into the device. A more proactive solution would involve setting thresholds and sending emails to admins, but 1) there are already plenty of scripts that do that, and 2) that is not a luxury I have on my sensors. I have inbound ssh, outbound 80 for updates and outbound 443 for logging.

Nevertheless, this reactive approach is much cooler than nothing at all and I think it would still be helpful on any snort installation no matter what other active health monitoring is in place.

Display most recent snort signatures on login

The first function I added to .bash_profile is called checksnortsigs(). It sorts the files in /etc/snort/rules by date order, and grabs the date of the most recent .rules file. It's that simple. It then just prints that information when you log in and gives a little advice on what to do next:

checksnortsigs()
{
if [ -f /etc/init.d/snortd ]; then
LATESTRULE=`ls -ltr /etc/snort/rules/*.rules tail -1 awk '{print $6, $7}'`
echo "-------------- Snort Installation Detected -----------------"
echo "The most recent snort rules on this machine were updated on:"
echo " ******* $LATESTRULE *******"
echo "If the date above is more than 1 month old, run oinkmaster"
echo "manually and verify it completes without error."
echo "------------------------------------------------------------"
echo
fi
}

checksnortsigs


Output (which is displayed as soon as the user logs in) looks like this:

Last login: Thu May 29 16:27:36 2008 from xxxxxxxx
-------------- Snort Installation Detected -----------------
The most recent snort rules on this machine were updated on:
******* May 30 *******
If the date above is more than 1 month old, run oinkmaster
manually and verify it completes without error.
------------------------------------------------------------

Display % dropped packets and Mbps stats on login

Shortly after that, and this came to me after playing around with sguil and seeing how nicely the snort.stats information is integrated into the analyst console, I decided that I also wanted to display recent % dropped packets and Mbps statistics each time someone logged in. There are a few more steps to get this one working, but they are all very easy:

1) Enable the following line in snort.conf:

preprocessor perfmonitor: time 300 file /var/log/snort/snort.stats pktcnt 10000

2) Restart snort

3) I created a very simple bash script that is basically one line of code along with a bit of "usage" code to make it easier for others to run. I called it get-snort-stats.sh. I created it for the bash_profile script, but it can be used as a standalone program at all. Here it is:

#!/bin/bash
# A very simple utility that will display the % dropped packets as well as throughput statistics.
# Snort records this information every 5 minutes
# Author: Seth Art
# Created: May 20th, 2008

###########################
#Usage
###########################
if [ -z $1 ]; then

echo
echo "Usage: get-snort-stats [number of lines to display]..."
echo exit
fi

case $1 in
'-h''--help')
echo
echo "Usage: get-snort-stats [number of lines to display]..."
echo " -h, --help display this help and exit"
echo
exit 1
;;esac

###########################
#Main
###########################
tail -$1 /var/log/snort/snort.stats awk -F , '{print "Dropped Packets = " $2, "\t", "Mbps = "$3}'



4) This bit ties the get-snort-stats command in with the .bash_profile script. Add the following function to .bash_profile

getsnortstats()
{
if [ -f /etc/init.d/snortd ]; then
echo "------------------------------------------------------------"
echo" Snort % Pkts dropped and mbits/sec for the last 20 minutes "
/usr/local/bin/get-snort-stats.sh 4
echo "------------------------------------------------------------"
fi
}

5) Add the call to the getsnortstats() function right below the checksnortsigs() fucntion call in bash_profile:

checksnortsigs
getsnortstats


5) Now I'm positive there is a better way to do this, but to make sure the snort.stats file doesn't grow out of control I simply put a line that rm's snort.stats every night in the same script I ued to run oinkmaster, recreate sig-msg.map, and restart snort. Not the most elegant solution I know, but it works...

When all is said and done, you should see the following information when you log in from now on:

Last login: Thu May 29 16:27:36 2008 from xxxxxxxx
-------------- Snort Installation Detected -----------------
The most recent snort rules on this machine were updated on:
******* May 30 *******
If the date above is more than 1 month old, run oinkmaster
manually and verify it completes without error.
------------------------------------------------------------
------------------------------------------------------------

Snort % Pkts dropped and mbits/sec for the last 20 minutes
Dropped Packets = 0.000 Mbps = 4.672
Dropped Packets = 0.000 Mbps = 4.796
Dropped Packets = 0.000 Mbps = 4.369
Dropped Packets = 0.000 Mbps = 5.071
------------------------------------------------------------

Enjoy :)

4 comments:

Leon Ward said...

Just a heads up, but I can see you being bitten by your last 20 minutes logic on your drop rate script.

Remember that perfmon only writes a log entry when x packets are seen in a specified time period. This means that if someone unplugs your tap or reconfigures your SPAN port you don't get any new entries in your stats file. Your script would make it look like the system is still functioning fine.

A better approach would be to pay attention to the time-stamp on each line.

-Leon

Seth said...

Thanks for the heads up. You are right, I did completely forget about that. Luckily the fact that I am deleting the snort.stats file every night should help identify any issues like a reconfigured SPAN or broken tap, however it is definitely something I want to look more closely at.

I wonder if changing the output to "Last 4 entries in snort.stats:", and then just including the timestamp on each line of output would be enough. Either way, it is something to look in to...

Thanks!

Leon Ward said...

Or somethine like

[22:08:57]lward@drax/var/tmp$ ./test.sh
Performance 26.63 minutes ago - 0.690 Mbps
Performance 20.56 minutes ago - 1.133 Mbps

[22:08:58]lward@drax/var/tmp$ cat ./test.sh
#!/bin/bash

FILENAME="now"
ENTRIES="2"
NOW=$(date +%s)

for line in `tail -n $ENTRIES $FILENAME`
do
TSTAMP=$(echo $line | awk -F , '{print $1}')
MBPS=$(echo $line |awk -F , '{print $3}')

TIMEDELTA=$(echo "scale=2 ; ($NOW - $TSTAMP) / 60" | bc -l)
echo Performance $TIMEDELTA minutes ago - $MBPS Mbps
done
[22:09:25]lward@drax/var/tmp$

Seth said...

Thanks again Leon. That looks like a great idea...