It's been over four months since I last tossed some digital ink on the blog. Rather than maintaining the pulse on latest and greatest in tech and dabbling in some web development, I've been concentrating on increasing my knowledge in personal finance and investing. Breathe, yes, there you go. I realize that it may come across as quite shocking that I'd actual diversify my reading interests in that area, but it's a nice change of pace. You might see a few blogs related to that in the future, but for now, we are going to keep this one geeky.

The focus of this entry will be on creating a bash script that automatically deletes files older than a certain number of days along with some bells and whistles to make the script flexible.

History

Given that my reputation for home network disasters, it won't be surprising to hear that I now ensure data from all my systems is backed up in at least 2-3 places at any given point in time. I even backup my data periodically to DVDs and keep them in a fireproof safe.

Laptop Backup

Let's talk a little about my backup strategy. First, my personal laptop. It backs up to an external hard drive that implements PGP's full disk encryption. Since my laptop is also using full disk encryption (not File Vault), that means that my data is backed up in two places at any given time. I also backup certain files on my laptop to my file server just to add another layer of comfort, which allows me to sleep at night.

Server Backups

As for my servers, well, I have three servers. Phobos (file server), cerberus (router), and icepick (web server). Every night, each server backups important configuration files into a gzipped file to a certain directory. For simplicity, this does not include other files that need backed up such as music, photos, etc.

I'm using the following naming convention for the backup filenames.

YYMMDD-<hostname>-backup.gzip.gpg

For example, here's what a backup file may look like

20081109-phobos-backup.gzip.gpg

Using this strategy, my goal is to maintain up to a year of backups for important files on each server. Once a day, these files are also synchronized to a central location on my file server, but that's outside the scope of what we are talking about here.

Problem and Requirements

The problem with this strategy is that periodically I need to manually delete files older than 365 days, which can be quite annoying. Over the weekend, I decided to write a script that will automatically delete backup files older than a certain number of days. Specific requirements included:

• User must be able to configure various options such as maximum number of days to retain files, log file location, email address for mailing reports, etc
• Script must support a basic testing  mode that will create a series of files in a test directory and verify things are working properly
• Script must generate a report containing the results of the script along with errors and mail it to a user configurable email address.
  
• Must prevent the script from deleting a large number of files accidently due to issues like files that all have the same mtime.

AutoBackupExpunger v1.0

Given those requirements and a few code snippits from AutoMySQLBackup, I think I've met most of them in AutoBackupExpunger v1.0. Included below is a copy of the script, please feel free to use or tweak it to meet your needs. By default, it probably won't work for you, but with some adjustments it should work just fine. Consider this "as-is" software and use at your own risk. I am not a bash guru by any stretch of the imagination. If you have suggestions for improving the script, please feel free to comment.

#!/bin/bash
#
# AutoBackupExpunger VER 1.0
#
# This program safely deletes daily backups based on user defined
# threshold of days. By default, it will only delete backup files
# older than 365 days. Since the script relies on the mtime of
# files within the directory, the script also has some checks to
# ensure that a large number of files aren't deleted by accident
# (e.g., all files have the same mtime). 
#
# ===============================================================

# ========================================
# User Defined Configuration Variables
# ========================================

# Date format for email messages
DATE=`date +%Y-%m-%d_%Hh%Mm`

# The number of days to retain backup files
MAX_DAYS=365

# The maximum number of files that the script will delete before
# generating an error message warning the user.
SAFETY_LIMIT=25

# Directory location of the backup files
BACKUP_DIRECTORY="/backup/archive"

# ========================================
# Advanced Configuration Variables
# ========================================

# Version of this software
VERSION=1.0

# Hostname of the system
HOST=`hostname`

# Location of the logfile
LOGFILE=/backup/$HOST-`date +%N`.log               # Logfile Name
LOGERR=/backup/ERRORS_$HOST-`date +%N`.log         # Logfile Name

# Mail setup
EMAIL="email@domain.com"

# ======================================================
# IO redirection for logging.
# ======================================================
touch $LOGFILE
exec 6>&1           # Link file descriptor #6 with stdout.
                    # Saves stdout.
exec > $LOGFILE     # stdout replaced with file $LOGFILE.
touch $LOGERR
exec 7>&2           # Link file descriptor #7 with stderr.
                    # Saves stderr.
exec 2> $LOGERR     # stderr replaced with file $LOGERR.

# ======================================================
# TESTING ONLY
# Creating test files (Only uncomment if you want to
# create test files in the testing directory. These
# values override your defaults)
# ======================================================
#BACKUP_DIRECTORY="/backup/archive_test"
#MAX_DAYS=5
#SAFETY_LIMIT=25
#FILES="*"
#COUNTER=0
#FILENAME_PREFIX=`date +%Y%m%d`

# Create testing directory if it doesn't already exist
#if [ ! -d $DIRECTORY ]
#then
#    echo Directory does not exist. Creating test_directory.
#    mkdir $DIRECTORY
#fi

# Loop through and create the test directories
#while [  $COUNTER -lt 25 ]; do
#    FILENAME_PREFIX=`date --date="$COUNTER days ago" +%Y%m%d`
#    touch -d $FILENAME_PREFIX $DIRECTORY/$FILENAME_PREFIX-testfile.txt
#       let COUNTER=COUNTER+1
#done


# =========================
# Begin execution of script
# =========================

echo ======================================================================
echo AutoBackupExpunger Files VER $VERSION
echo
echo Expunging Old Backup Files - $HOST
echo ======================================================================
echo Backup Expunger Start Time `date`
echo ======================================================================

# Count all files that would get deleted
NUM_FILES=`find $BACKUP_DIRECTORY -type f -mtime +$MAX_DAYS -print | wc -l`

if (( $NUM_FILES == 0 ))
then
    echo
    echo There are $NUM_FILES files that are older than $MAX_DAYS days.
else
    echo
    echo There are $NUM_FILES files that will be deleted because they are older than $MAX_DAYS days.
fi

# Delete files that are safe to delete.
if (( $NUM_FILES > 0 && $NUM_FILES < $SAFETY_LIMIT ))
then
    echo
    echo Deleting backup files older than $MAX_DAYS days...
    echo
    for FILE in `find $BACKUP_DIRECTORY -type f -mtime +$MAX_DAYS`
    do
            echo $FILE
#        rm $FILE
    done
    echo
    echo All files successfully deleted.
elif (( $NUM_FILES == 0 ))
then
    echo
    echo There are no files that need rotated.
else
    echo
    echo The backup expunger failed because the number of files that would have been deleted exceed the threshold you defined. A separate email message was sent notifying the administrator of this error.
        ERROR_MSG="The backup expunger script attempted to delete $NUM_FILES files. This exceeded the maximum limit of files that can be deleted, which is set to $SAFETY_LIMIT. This is a safety measure to ensure that all your precious backups don't go bye bye. Therefore, the backup rotation failed. You need to check this out."
        echo $ERROR_MSG | mail -s "Auto Backup Expunger Max Files Error for $HOST - $DATE" -t "$EMAIL"
fi

echo
echo ======================================================================
echo Backup Expunger End `date`
echo ======================================================================

# Clean up IO redirection
exec 1>&6 6>&-      # Restore stdout and close file descriptor #6.
exec 1>&7 7>&-      # Restore stdout and close file descriptor #7.

# Send email notifications
cat "$LOGFILE" | mail -a "From: $HOST Cron Job <$EMAIL>" -s "Auto Backup Expunger Log for $HOST - $DATE" $EMAIL
if [ -s "$LOGERR" ]
then
    cat "$LOGERR" | mail -a "From: $HOST Cron Job <$EMAIL>" -s "ERRORS REPORTED: Auto Backup Expunger Error Log for $HOST - $DATE" $EMAIL
fi

# Set status condition
if [ -s "$LOGERR" ]
then
        STATUS=1
else
        STATUS=0
fi

# Clean up log files
eval rm -f "$LOGFILE"
eval rm -f "$LOGERR"

# Exit the script
exit $STATUS