Wednesday, June 11, 2008

Mysql fix

I don't believe this post will be very useful to anyone else, but I want to record it anyway. I noticed a few weeks ago that my drupal installation was complaining that the watchdog table had crashed. With my limited understanding of mysql, I didn't event know that a table *could* crash. Everything else on the site looked fine to the anonymous user so I just ignored it.



That brings me to today. I found this interesting script online that will dump all of my mysql databases every hour to another file system. I figured I would give a shot. I entered my root db password and the dst directory and let her rip. I got a few errors right away:



[root@www storage]# mysql-backup.sh

mysqldump: Error 1194: Table 'watchdog' is marked as crashed and should be repaired when dumping table `watchdog` at row: 283

mysqldump: Got error: 145: Table './drupal/watchdog' is marked as crashed and should be repaired when using LOCK TABLES

mysqldump: Got error: 145: Table './gallery2/g2_CacheMap' is marked as crashed and should be repaired when using LOCK TABLES



Google quickly found the following post: http://gallery.menalto.com/node/72721, where a user kindly posted the solution to their own problem:



Stage 1: Checking your tablesRun myisamchk *.MYI or myisamchk -e *.MYI if you have more time. Use the -s (silent) option to suppress unnecessary information.If the mysqld server is stopped, you should use the --update-state option to tell myisamchk to mark the table as “checked.”You have to repair only those tables for which myisamchk announces an error. For such tables, proceed to Stage 2.If you get unexpected errors when checking (such as out of memory errors), or if myisamchk crashes, go to Stage 3.
Stage 2: Easy safe repairFirst, try myisamchk -r -q tbl_name (-r -q means “quick recovery mode”). This attempts to repair the index file without touching the data file. If the data file contains everything that it should and the delete links point at the correct locations within the data file, this should work, and the table is fixed.source:
Website



[root@www storage]# myisamchk -r -q /var/lib/mysql/drupal/watchdog.MYI

- check record delete-chain

- recovering (with sort) MyISAM-table '/var/lib/mysql/drupal/watchdog.MYI'Data records: 513

- Fixing index 1

Found block that points outside data file at 122592

MyISAM-table '/var/lib/mysql/drupal/watchdog.MYI' is not fixed because of errors

Try fixing it by using the --safe-recover (-o), the --force (-f) option or by not using the --quick (-q) flag

[root@www storage]# myisamchk -r /var/lib/mysql/drupal/watchdog.MYI

- recovering (with sort)

MyISAM-table '/var/lib/mysql/drupal/watchdog.MYI'

Data records: 513

- Fixing index 1

Found block that points outside data file at 122592

- Fixing index 2



As you can see, I had to get rid of the -q option to get it to work, but it did in fact work. Same command worked to fix g2_CasheMap, but that one took quite a bit longer.


Looks like that did the trick.

Friday, June 6, 2008

Some snort login kung-fu...

I was recently playing around with my .bash_profile file looking for new ways to alert myself as well as my team to problems with production snorts. I ended up with two little tricks that I have found really useful and I figured I would share.

For those that don't know, the .bash_profile file is an sh script that runs at user login. At a bare minimum it sets the users PATH, but it can be used for a whole lot more. It's located in the root of the users home directory. Ex: /home/snort/.bash_profile, or /root/.bash_profile

Before I go any further I will tell you that both of these tricks are obviously reactive in nature. They only let you know there is a problem the next time you log into the device. A more proactive solution would involve setting thresholds and sending emails to admins, but 1) there are already plenty of scripts that do that, and 2) that is not a luxury I have on my sensors. I have inbound ssh, outbound 80 for updates and outbound 443 for logging.

Nevertheless, this reactive approach is much cooler than nothing at all and I think it would still be helpful on any snort installation no matter what other active health monitoring is in place.

Display most recent snort signatures on login

The first function I added to .bash_profile is called checksnortsigs(). It sorts the files in /etc/snort/rules by date order, and grabs the date of the most recent .rules file. It's that simple. It then just prints that information when you log in and gives a little advice on what to do next:

checksnortsigs()
{
if [ -f /etc/init.d/snortd ]; then
LATESTRULE=`ls -ltr /etc/snort/rules/*.rules tail -1 awk '{print $6, $7}'`
echo "-------------- Snort Installation Detected -----------------"
echo "The most recent snort rules on this machine were updated on:"
echo " ******* $LATESTRULE *******"
echo "If the date above is more than 1 month old, run oinkmaster"
echo "manually and verify it completes without error."
echo "------------------------------------------------------------"
echo
fi
}

checksnortsigs


Output (which is displayed as soon as the user logs in) looks like this:

Last login: Thu May 29 16:27:36 2008 from xxxxxxxx
-------------- Snort Installation Detected -----------------
The most recent snort rules on this machine were updated on:
******* May 30 *******
If the date above is more than 1 month old, run oinkmaster
manually and verify it completes without error.
------------------------------------------------------------

Display % dropped packets and Mbps stats on login

Shortly after that, and this came to me after playing around with sguil and seeing how nicely the snort.stats information is integrated into the analyst console, I decided that I also wanted to display recent % dropped packets and Mbps statistics each time someone logged in. There are a few more steps to get this one working, but they are all very easy:

1) Enable the following line in snort.conf:

preprocessor perfmonitor: time 300 file /var/log/snort/snort.stats pktcnt 10000

2) Restart snort

3) I created a very simple bash script that is basically one line of code along with a bit of "usage" code to make it easier for others to run. I called it get-snort-stats.sh. I created it for the bash_profile script, but it can be used as a standalone program at all. Here it is:

#!/bin/bash
# A very simple utility that will display the % dropped packets as well as throughput statistics.
# Snort records this information every 5 minutes
# Author: Seth Art
# Created: May 20th, 2008

###########################
#Usage
###########################
if [ -z $1 ]; then

echo
echo "Usage: get-snort-stats [number of lines to display]..."
echo exit
fi

case $1 in
'-h''--help')
echo
echo "Usage: get-snort-stats [number of lines to display]..."
echo " -h, --help display this help and exit"
echo
exit 1
;;esac

###########################
#Main
###########################
tail -$1 /var/log/snort/snort.stats awk -F , '{print "Dropped Packets = " $2, "\t", "Mbps = "$3}'



4) This bit ties the get-snort-stats command in with the .bash_profile script. Add the following function to .bash_profile

getsnortstats()
{
if [ -f /etc/init.d/snortd ]; then
echo "------------------------------------------------------------"
echo" Snort % Pkts dropped and mbits/sec for the last 20 minutes "
/usr/local/bin/get-snort-stats.sh 4
echo "------------------------------------------------------------"
fi
}

5) Add the call to the getsnortstats() function right below the checksnortsigs() fucntion call in bash_profile:

checksnortsigs
getsnortstats


5) Now I'm positive there is a better way to do this, but to make sure the snort.stats file doesn't grow out of control I simply put a line that rm's snort.stats every night in the same script I ued to run oinkmaster, recreate sig-msg.map, and restart snort. Not the most elegant solution I know, but it works...

When all is said and done, you should see the following information when you log in from now on:

Last login: Thu May 29 16:27:36 2008 from xxxxxxxx
-------------- Snort Installation Detected -----------------
The most recent snort rules on this machine were updated on:
******* May 30 *******
If the date above is more than 1 month old, run oinkmaster
manually and verify it completes without error.
------------------------------------------------------------
------------------------------------------------------------

Snort % Pkts dropped and mbits/sec for the last 20 minutes
Dropped Packets = 0.000 Mbps = 4.672
Dropped Packets = 0.000 Mbps = 4.796
Dropped Packets = 0.000 Mbps = 4.369
Dropped Packets = 0.000 Mbps = 5.071
------------------------------------------------------------

Enjoy :)

Pentest Home Lab - 0x2 - Building Your AD Lab on Premises

In Pentest Home Lab - 0x0 - Building a virtual corporate domain , we talked about why you would want to build your own AD pentest lab, where...