I was only 6 when Linus shared his first offering to the world, but I do remember the linux superbowl ad by IBM.

You’re right… I probably could have typed

nslookup targetNode

60 times. But I am a programmer! SO I decided to pass my choir off to a shell script.

If you have a similar need, read on!

The script is simple, just add all target nodes as a space separated list, and run the file.

#!/bin/bash 
# author: Edward A Webb (http://edwardawebb.com
# license: Just use it! its like 10 lines of shell script...
 
# add your actual nodes here.....
nodes=( gonzo fenway einstein yogurt toxic babmbi rambo donna INXS etc... )
 
ips=( ) # placeholder for results
 
printf "\t Determing IP addresses for provided nodes \n ignore these lines...\n"
for (( i = 0 ; i < ${#nodes[@]} ; i++ ))
do
	str=$(nslookup ${nodes[$i]} | tail --lines=2 | head --lines=1)
	ips[$i]=$(echo $str | cut -f 2 -d " ")
 
done
 
printf "\t COMPLETE\n Here are the results you care about..\n"
 
for (( i = 0 ; i < ${#ips[@]} ; i++ ))
do
	echo address for ${nodes[$i]} is ${ips[$i]} | tee -a ip_results.txt
done

The bash script included after the break reads all directories into an array that we can loop through and manipulate as needed.

Background

My webroot is pretty straightforward. Each subdirectory is the full URL of a site. Example:

webroot
|-- baungenjar.com
|-- bythebootstrap.com
|-- cafe--espresso.com
|-- crm.webbmaster.org
|-- eaw-technologies.com
|-- edwardawebb.com
|-- entrepreneurmeetcapitalist.com
|-- kicksareforribs.com
|-- ollitech.com
|-- outlaw-photography.com
|-- piwik.baungenjar.com
|-- rendinaro.com
|-- riversidegrillnh.com
|-- sidi.webbmaster.org
|-- status.webbmaster.org
|-- steadfastsites.com
|-- stonybrookpottery.com
|-- survey.webbmaster.org
|-- top.webbmaster.org
|-- trippymedia.com
|-- webbmaster.org
`-- wishlist.webbmaster.org

So I want to take each directory and create a gzipped tarball using tar -zcf.    I don’t want to do any manual intervention.

*ALso, I want to exclude some directories.

Solution

This actually took me much longer than it should have. Too many articles insisted you should change the internal field separator (IFS) to a newline character and just use ls.  Now in my training I was learn-ed that messing with IFS can be dangerous, and will disrupt many things, including any command that relies on options or arguments (i.e. ALL)
So rather than make my entire script confirm to pesky newlines returned by ls, I did one simpler – eliminate the newlines in place of a decent token separator, like a space!

To exclude directories I just use a magic file “.DONT_BACKUP” that the script checks for.

Code

#!/bin/bash
#
# Backup all directories within webroot
# use empty file ".DONT_BACKUP" to exclude any directory
 
 
# days to retain backup. Used by recycler script
DEFRETAIN=14
LOGFILE=/home/webb_e/site_backups/WebrootBackup.log
#
#
BU_FILE_COUNT=0
#
# and name of backup source subfolder under the users home
WEBDIR=webroot
#
# and name of dest folder for tar files
DESDIR=site_backups
 
#alright, thats it for config, the rest is script
#########################################
 
 
cd ${HOME}/${WEBDIR}/
 
 
TODAY=`date`
BU_FILE_COUNT=0
suffix=$(date +%m-%d-%Y)
printf "\n\n********************************************\n\tSite Backup r Log for:\n\t" | tee -a $LOGFILE
echo $TODAY | tee -a $LOGFILE
printf "********************************************\n" $TODAY | tee -a $LOGFILE
echo "see ${LOGFILE} for details"
 
#for DIR in $(ls | grep ^[a-z.]*$) 
 
for DIR in $(ls | grep ^[a-z.]*$) 
do
	echo $DIR
	#tar the current directory
	if [ -f $DIR/.DONT_BACKUP ]
	then
 
		printf "\tSKIPPING $DIR as it contains ignore file\n" | tee -a $LOGFILE
 
	else
		cpath=${HOME}/${DESDIR}/${DIR}
		#
		#check if we need to make path
		#
		if [ -d $cpath ]
		then
			# direcotry exists, we're good to continue
			filler="umin"
		else
			echo Creating $cpath
			mkdir -p $cpath
			echo $DEF_RETAIN > $cpath/.RETAIN_RULE
		fi
		#
 
		tar -zcf ${HOME}/${DESDIR}/${DIR}/${DIR}_$suffix.tar.gz ./$DIR
		BU_FILE_COUNT=$(( $BU_FILE_COUNT + 1 ))
	fi
 
done
printf "\n\n********************************************\n" | tee -a $LOGFILE
echo $BU_FILE_COUNT sites were backed up
printf "********************************************\n" $TODAY | tee -a $LOGFILE

Result

$ ~/scripts/file_backups/SiteBackup.sh
see /home/myself/site_backups/WebrootBackup.log for details

********************************************
        Site Backup r Log for:
        Thu Apr 15 14:31:53 PDT 2010
********************************************
albums.stonybrookpottery.com
baungenjar.com
bythebootstrap.com
crm.webbmaster.org
digbiz.webroot.com
dotproject.webbmaster.org
edwardawebb.com
kicksareforribs.com
mantis.webbmaster.org
        SKIPPING mantis.webbmaster.org as it contains ignore file
ollitech.com
openx.webbmaster.org
piwik.baungenjar.com
rendinaro.com
riversidegrillnh.com
sidi.webbmaster.org
status.webbmaster.org
steadfastsites.com
stonybrookpottery.com
survey.webbmaster.org
svn.webbmaster.org
taskfreak.webbmaster.org
test.kicksareforribs.com
texpat.webbmaster.org
top.webbmaster.org
        Creating /home/myself/site_backups/top.webbmaster.org
trac.webbmaster.org
trippymedia.com
wave.webbmaster.org
webbmaster.org
wishlist.webbmaster.org

********************************************
33 sites were backed up
********************************************

But wait! We wanted to automate this whole thing right? And so we shall.

Using Cron to automate the process

If your using a webhost they likely provide a GUI to add cron jobs. If that’s the case you can just point to the full path where you saved the above script, select the interval and your good to go.

If your using this on your own server you’ll need to get your hands dirty with a crontab. You can open the crontab file in your editor of choice, or call it from the command line. IN this example we’ll rely in vi, my systems default editor.
Create a crontab file if it does not already exist and open it for edit

crontab -e

You may see some existing lines or you may not. Just remember one job per line. THe layout may seem overwhelming at first, but its quite simple, and breaks down like this

min     hour     day    month    weekday     job_to_Run

The values are in the respective ranges for day of week 0 is Sunday.

0-59    0-23     1-31    1-12     0-6        filename

To omit a field replace it with an asterisk (*) which means all values. Alternately you may use comma separated lists. Although I believe it will treat any whitespace as a delimiter I use tabs to make the organization a little nicer.

So let’s suppose I want to run this job nightly, it is after all named DAILY sql backup
I will add the following line to my crontab

15    0     *    *   *     $HOME/scripts/file_backup.sh > logfile.log

This means every day @ 00:15 a.k.a 15 minutes past midnight it will run the script and print any output into the specified logfile.
If you leave off the redirect to logfile it will email the user with the results. To omit any output use the handy standby

>/dev/null 2>&1

.

More Help

If you are curious about the script that will actually recycle old backups, then I suggest Log Recycler Script which could easily be updated to handle .tar.gz files instead of .sql.gz files

If you want to backup your MySQL databases, I have that too, Shell script to backup multiple databases

IMpossible you say? Not with the release of Apache 2.2.13!

I stumbled on this nice little feature called SNI (or Server Name Indication) that allows multiple domains to share an IP and implement SSL without showing warnings to users or confusing Apache.  I found this beauty after reading another great article on Linux Magazine, Ten Things You Didn’t KNow Apache (2.2) Could Do

So those of you running your own servers can take advantage by upgrading today!

For those of you relying on a Host you should start bombarding them with requests immediately. DreamHost members can vote up the already created suggestion  to implement this.

So my goal in the article that follows is to setup a powerful workstation that includes Cygwin, PuTTY, and WinSCP.  WinSCP is not a necessary tool, but I’ll admit I like to resort to a graphical SCP tool every now and again.  Unlike a few other articles I have read I will not encourage you to use passphrase-less keys, just too risky in my mind.  Instead we will just limit the number of times we need to enter that password to say about…. 1.

To recap, our Goals are;

• Install powerful windows based tools. (Cygwin, PuTTY and WinSCP)
• Generate a pair of 2048 bit RSA keys (With a passphrase)
• Disseminate the public key to all the nodes we know (or connect to)
• Employ Pageant and SSH-Agent to limit the need to enter that single passphrase.

So let’s get started.

Installing the Tools

Most anyone who works with Unix nodes from a window’s desktop knows the joy of PuTTY. Perhaps a few less know about Cygwin.  These tools are a must have in any IT toolbox.

Cygwin

Cygwin, available here, allows Unix like commands to run in a windows command console.   When installing the utility we need to make sure to check off a few extras that are not part of the default install.

Kick off the install until you get to a list of the components to install. Expand the Net category and check off OpenSSH and OpenSSL.
 Let the installation complete normally otherwise.

PuTTY

You know all about PuTTY I am sure.  My only real advice here is to download the complete PuTTY suite of tools with the installer. There are two main perks to this.  1) You get all the goodies like PuTTYGen and Pageant. 2) If your running Vista you won’t get those annoying UAC alerts every time you kick it off.

You can find the PuTTY suite here. Be sure to grab the one listed under “A Windows installer for everything except PuTTYtel.”

Don’t worry, if you already had the stand alone utility installed, it will preserve all yoru settings so long as you install to the same folder.

WinSCP

WinSCP is a graphical interface for moving files between your local machine and remote servers.  If you want to stay true to your terminals you can just stick with Cygwin and the SCP utility (which we’ve already installed).

Grab WinSCP from here, no special install needed.

Generating our Keys

OK!  So now we can connect to our remote servers any way conceivable to man. Well maybe that’s an over statement, but I think our bases are pretty well covered.

The next hurdle is to create a pair of digital keys.  We keep one key (the private key) on our local machine, and never share it with anyone.  We disseminate (or distribute) the other key (public) out to any server we want to connect to.  The key pair will be reunited on connection allow us to authenticate without entering our password for the remote server.

First open up Cygwin.  Ahh isn’t she beautiful.

Type ssh-keygen -t rsa at the prompt and follow the queues entering information.  Be sure to enter a passphrase! You may accept the default path for a key location.

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/Eddie/.ssh/id_rsa):
Created directory '/home/Eddie/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/Eddie/.ssh/id_rsa.
Your public key has been saved in /home/Eddie/.ssh/id_rsa.pub.
The key fingerprint is:
74:17:ef:24:e6:5f:1b:f3:e0:6a:1b:fe:ae:a6:fe:24 Eddie@HP-dv6z-420
The key's randomart image is:
+--[ RSA 2048]----+
|  .-o.           |
|+ .o*  .         |
|+F .o . o        |
|..o  + .         |
|  -     S        |
|   -             |
|   +-  F  o      |
|.  .G . .        |
| o+*+o     o     |
+-----------------+

Now since you should be in your home directory you can type  ls -a and you should notice a new directory .ssh inside their will be your public (id-rsa.pub) and private (id-rsa) keys. If you chose an alternate path while generating the keys, be sure to move the private key into this folder.

Disseminate our Public Key

We’ll stick with Cygwin for a bit longer and use it’s scripting abilities to share our public key with any servers we want to connect to. If you only have one remote server in mind than a script is probly overkill, but you will want to follow the steps within the script. In my case I have about 20 servers, so I’ll take all the automation I can get.

The Dissemination Scripts

There are two scripts. One runs locally, and the other gets sent over SCP and then runs as our remote henchmen before being purged like a failed hitman.

Script 1 – Called Locally 

#! /bin/bash
# Author: Eddie Webb - edwardawebb.com
 
# Deploys a public SSH key to multiple remote nodes. It will convert from Putty Format
 
# Calls Script:
#	addPublicKeyRemotely.sh
 
 
 
 
## Check arghuments
if [ $# -ne 1 ]
then
	printf "Usage: \n %s PuttyPubKey\n\n" `basename $0`
fi
 
 
#create a log file
LOGFILE=Results.log
 
 
 
SERVERS=( server1 server2 funnyname anotherserver )
#NOte: if your username is different on each node - make an array below that correspoinds to the node list above
USER=YOUREMOTEUSERNAME
 
SSHKEY=$1
SSHKEYBASE=`basename $1`
SCRIPTPATH=`dirname $0`
 
## loop thorugh all servers sending key and remote script - purges afer script runs.
for (( i = 0 ; i < ${#SERVERS[@]} ; i++ ))
do
	echo Sending to Host ${SERVERS[$i]}
	scp $SSHKEY $SCRIPTPATH/addPublicKeyRemotely.sh $USER@${SERVERS[$i]}:~/
	ssh $USER@${SERVERS[$i]} "~/addPublicKeyRemotely.sh $SSHKEYBASE;rm -f ~/addPublicKeyRemotely.sh" >> $LOGFILE
	if [ $? -eq 0 ]
	then
		echo Key successfully pushed to ${SERVERS[$i]} >> $LOGFILE
		echo SUCCESS
	else
		echo FAILED pushing key to ${SERVERS[$i]} >> $LOGFILE
		echo FAILED
	fi
	printf "\n\n\n"
done

Script 2 – Called remotely – Name important (addPublicKeyRemotely.sh)

#!/bin/bash
# Author: Eddie Webb - edwardawebb.com
 
 
# Called by Script:
#		promotePublicKey.sh
 
# Arguments:
#		name of converted SSH key 
 
# this file is uplaoded to remote serveres and called to handle remote work before being deleted.
 
 
	if [ ! -d .ssh ]
	then
		mkdir -p .ssh
	fi
	touch .ssh/authorized_keys
	chmod 0600 .ssh/authorized_keys
	cat $1 >> .ssh/authorized_keys
	rm -f $1

Run the first script specifying your PUBLIC key as the only parameter.
Now the downside is that you will still need to enter your remote passwords for each server, but hey it will be the last time.
The result should look something like;

 
$ scripts/promotePublicKey.sh .ssh/id_rsa.pub
Sending to Host domain.com
YOURUSER@domain.com's password:
id_rsa.pub                                    100%  399     0.4KB/s   00:00
addPublicKeyRemotely.sh                       100%  387     0.4KB/s   00:00
YOURUSER@domain.com's password:
SUCCESS

Testing Cygwin

You should now be able to make a remote connection entering only your new key passphrase(the passwordless part is yet to come).

$ ssh YOURUSER@domain.com
Enter passphrase for key '/home/Eddie/.ssh/id_rsa':

Don’t worry, you won’t always enter the passphrase, that was a test. Be patient with me

Importing to PuTTY

Putty uses its own format for private keys (PPK) so we need to import our openSSH private key and convert it to a PPK for Putty to use. We’ll do this without overwritting the private key we have (still needed by Cygwin).

Open up PuTTYgen from your start menu.

First Load your existing private key (C:\cygwin\home\<YOURNAME>\.ssh\id-rsa)

Import existing key to PuTTYge

 After you import the key you will be asked for your key’s passphrase before seeing the screen below. It lists the details about your public key.

The imported key in PuTTYgen

Finally click the Save private key button (highlighted in Yellow)  this will save the priavte key as a PPK file. I typically save this in my PuTTY installation directory under a folder names Keys. Do not overwrite your existing key, and keep the PPK extension. 

Repetitive Entry Avoidance (A.K.A. Enter the passphrase once, and only once)

Now we need to modify PuTTY and Cygwin to keep our keys in memory so we only enter the passphrase once per session.

Using Pageant

Pageant, which installed with PuTTY, will hold the keys open for PuTTy and WinSCP.

Find the shortcut to Pageant in your start menu. Make a copy of the shortcut and paste it into your Startup menu. This will kick off pageant every time windows boots. However, we need to tell the shortcut to load our private key.

Now right-click that icon and select properties. 
Edit the entry that says Target and change it from;

“C:\Program Files (x86)\PuTTY\pageant.exe”

To;

“C:\Program Files (x86)\PuTTY\pageant.exe” “C:\Program Files (x86)\PuTTY\Keys\id-rsa.ppk” 

Click Apply and then close the properties window.  

Now try clicking the icon. It should immediately ask you for your passphrase. This is the same prompt you will see next time you boot windows.

Go ahead an test out PuTTY now. You will not be prompted for a password, and instead will see something like’

Using username "YOURUSER".
Authenticating with public key "imported-openssh-key" from agent

Using SSH-Agent for Cygwin

Cygwin doesn’t use PuTTY’s ppk file, so we need a manner to retain the private id-rsa key within our Cygwin sessions.

Add the following script to your .bash_profile;

## only ask for my SSH key passphrase once!
#use existing ssh-agent if possible
if [ -f ${HOME}/.ssh-agent ]; then
   . ${HOME}/.ssh-agent > /dev/null
fi
if [ -z "$SSH_AGENT_PID" -o -z "`/usr/bin/ps -a|/usr/bin/egrep \"^[ ]+$SSH_AGENT_PID\"`" ]; then
   /usr/bin/ssh-agent > ${HOME}/.ssh-agent
   . ${HOME}/.ssh-agent > /dev/null
fi
ssh-add ~/.ssh/id_rsa

Save the file and hop back to Cygwin.  You can either restart Cygwin, or source the profile file to test this feature out. Since I left Cygwin open I’ll just source the file.

$ . .bash_profile
Enter passphrase for /home/Eddie/.ssh/id_rsa:
Identity added: /home/Eddie/.ssh/id_rsa (/home/Eddie/.ssh/id_rsa)

Now you SSH to eighty different servers and not enter the passphrase again!

Conclusion

We’re done! Thanks for bearing with my article, I know it was not too concise, but I hope you forged through it. If you have any questions, let me know.

It has also been pointed out as a way to maintain the schema.

Write a script to loop through all current tables and drop them? – NO!

You could get a listing of all tables, iterate through an loop and drop the tables, but let’s see if we can’t think of a one liner…

Rely on mysqldump and mysql to do all the work? – YES!

We need to drop tables, that’s obvious.  But how to get all the tables with drop command in hand.  What about our ever so faithful and functional friend mysqldump? Of course! Not only can mysqldump handle giving us the names of all the tables, but it can also hand us the drop command.

You’ll need to execute this command where mysqldump and mysql binaries live, typically this is /usr/bin.

/usr/bin/mysqldump -uuser_name -psecretpassword --add-drop-table database_name | grep ^DROP | /usr/bin/mysql  -uuser_name -psecretpassword database_name

SO we pump out all the table and data from the database,use grep to trash everything but the lines that begin with the drop commands and pass that right back into mysql. Works, but should we be passing and parsing all the data just to trash it? Probably not.

Keeping the workload small – no data!

Particularly for those of you that have rather large Databases we don’t want to waste time passing around data, so let’s leave it out entirely.

/usr/bin/mysqldump -uuser_name -psecretpassword --no-data --add-drop-table database_name | grep ^DROP | /usr/bin/mysql  -uuser_name -psecretpassword database_name

The problem

I have multiple databases that need to be backed up into different archive folders. Each database requires a unique host, username and password.

THe solution

The bash shell of course! First off we should all be familiar with mysqldump.  It lives idle in nearly every ‘nix box just waiting to dump a mysql database onto the screen, or file. ANd unless you want your mysqldump binary to be sad you should take full well use of it.

The most basic use of the program couldn’t be simpler, but it is happy to meet more complex needs as well. FOr this article we’ll keep it simple.

mysqldump -h HOSTNAME -u USERAME -pPASSWORD DATABASE > OUTPUTFILE

now you can of course add additional switches, c and e are two I use pretty frequently. See your machine’s man pages to learn more.

But wait that only handles 1 DB on 1 host with 1 user and password. True. SO here’s where we take use of a script to make this puppy work overtime.

The Shell Scripts

Many mysql Databases on one host

#!/bin/bash
 
#####
# Set these values!
####
 
 
# space seperated list of domain names (will be used as part of the output path)
domains=( name4subfolders inSQLfolderabove canbedomains orsomethingelse )
#list of corresponding DB names
sqldbs=( fulldbname1 db2 db3 db4 )
 
#list of IDs and passwords
usernames=( user1 user2 user3 user 4 )
passwords=( pass1 pass2 pass3 pass4 )
 
#Directory to save generated sql files (domain name is appended)
opath=$HOME/sql_dumps/
 
# your mysql host
mysqlhost=mysql.webbmaster.org
 
 
 
#####
# End of config values
#####
 
 
#date to append
suffix=$(date +%Y-%m-%d)
 
 
#run on each domain
for (( i = 0 ; i < ${#domains[@]} ; i++ ))
do
	#set current output path
	cpath=$opath${domains[$i]}
 
	#check if we need to make path
	if [ -d $cpath ]
	then
		# direcotry exists, we're good to continue
		filler="just some action to prevent syntax error"
	else
		#we need to make the directory
		echo Creating $cpath
		mkdir -p $cpath
	fi
 
	#now do the backup
	SQLFILE=${cpath}/${sqldbs[$i]}_$suffix.sql.gz
 
	mysqldump -c -h $mysqlhost --user ${usernames[$i]} --password=${passwords[$i]} ${sqldbs[$i]} 2>error | gzip > $SQLFILE
 
	if [ -s error ]
	then	   
		printf "WARNING: An error occured while attempting to backup %s  \n\tError:\n\t" ${sqldbs[$i]} 
		cat error
		rm -f er
	else
		printf "%s was backed up successfully to %s\n\n" ${sqldbs[$i]} $SQLFILE
	fi
done

In either case you’ll want to save it to a file, let’s say… daily_sql_backup.sh
And then make the script executable

#chmod 0750 daily_sql_backup.sh

Now you can test the script by calling it by name

# daily_sql_backup.sh

Sweet, a single command will backup any databases we include in the script. And we can call it again and again.

A simple output reports the results;

database1 was backed up successfully to /home/YOURNAME/sql_dumps/site1.com/database1_2009-01-14.sql.gz
 
taskfreak_database was backed up successfully to /home/YOURNAME/sql_dumps/taskfreak.sitetwo.com/taskfreak_database_2009-01-14.sql.gz
 
mantis_database was backed up successfully to /home/YOURNAME/sql_dumps/mantios.yetanother.org/mantis_database_2009-01-14.sql.gz

Error Reporting 

I recently had to update the error handling because it was letting failures pass by!

That was because mysqldump would fail, throw an error to stderr, but gzip would then come along and happily report that is successfully compressed nothing!

The new model outputs any errors to a file named “error” before everything is piped into gzip and the error code is lost. We then check for the file, and if present show it to the user (log it) before deleting the file and moving on.

So in a bad scenario I would now see:

Attempting to run MySQL dump on 2 databases
domain1.com
domain2.com

domain1_com was backed up successfully to /home/user/sql_dumps/
domain1/domain1_com_2011_05_05.sql.gz

WARNING: An error occured while attempting to backup domain2
        Error:
        mysqldump: Got error: 1045: Access denied for user 'domain2_com'@'domain2.com' (using password: YES) when trying to connect

But wait! We wanted to automate this whole thing right? And so we shall.

Using Cron to automate the process

If your using a webhost they likely provide a GUI to add cron jobs. If that’s the case you can just point to the full path where you saved the above script, select the interval and your good to go.

If your using this on your own server you’ll need to get your hands dirty with a crontab. You can open the crontab file in your editor of choice, or call it from the command line. IN this example we’ll rely in vi, my systems default editor.
Create a crontab file if it does not already exist and open it for edit

crontab -e

You may see some existing lines or you may not. Just remember one job per line. THe layout may seem overwhelming at first, but its quite simple, and breaks down like this

min     hour     day    month    weekday     job_to_Run

The values are in the respective ranges for day of week 0 is Sunday.

0-59    0-23     1-31    1-12     0-6        filename

To omit a field replace it with an asterisk (*) which means all values. Alternately you may use comma separated lists. Although I believe it will treat any whitespace as a delimiter I use tabs to make the organization a little nicer.

So let’s suppose I want to run this job nightly, it is after all named DAILY sql backup
I will add the following line to my crontab

15    0     *    *   *     $HOME/scripts/daily_sql_backup.sh > logfile.log

This means every day @ 00:15 a.k.a 15 minutes past midnight it will run the script and print any output into the specified logfile.
If you leave off the redirect to logfile it will email the user with the results. To omit any output use the handy standby

>/dev/null 2>&1

.

well I think that covers it. THere’s tons of good resources to learn more about any particular topic, but I would be happy to field comments.

The End Result

After your newly created cron jobs have the chance to run for a few days you’ll end up with a nice and neat directory structure like this;

sql_dumps/
|-- edwardawebb.com
|   |-- edwardawebb_wordpress_01-13-2009.sql.gz
|   |-- edwardawebb_wordpress_01-14-2009.sql.gz
|   |-- edwardawebb_wordpress_01-15-2009.sql.gz
|   |-- edwardawebb_wordpress_01-16-2009.sql.gz
|   |-- edwardawebb_wordpress_01-17-2009.sql.gz
|   |-- edwardawebb_wordpress_01-18-2009.sql.gz
|   `-- edwardawebb_wordpress_01-19-2009.sql.gz
|-- mantis.mainsite.org
|   |-- mainsite_mantis_01-13-2009.sql.gz
|   |-- mainsite_mantis_01-14-2009.sql.gz
|   |-- mainsite_mantis_01-15-2009.sql.gz
|   |-- mainsite_mantis_01-16-2009.sql.gz
|   |-- mainsite_mantis_01-17-2009.sql.gz
|   |-- mainsite_mantis_01-18-2009.sql.gz
|   `-- mainsite_mantis_01-19-2009.sql.gz
`-- taskfreak.mainsite.org
    |-- mainsite_taskfreak_01-11-2009
    |-- mainsite_taskfreak_01-11-2009.sql.gz
    |-- mainsite_taskfreak_01-12-2009.sql.gz
    |-- mainsite_taskfreak_01-13-2009.sql.gz
    |-- mainsite_taskfreak_01-14-2009.sql.gz
    |-- mainsite_taskfreak_01-15-2009.sql.gz
    |-- mainsite_taskfreak_01-16-2009.sql.gz
    |-- mainsite_taskfreak_01-17-2009.sql.gz
    |-- mainsite_taskfreak_01-18-2009.sql.gz
    `-- mainsite_taskfreak_01-19-2009.sql.gz

Note: this example would generate 3 files each night. After 1 month thats ~150 files depending on the month. That’s why I also wrote a simple Recycler script to purge all old files. As soon as I draft that article I’ll ad the link here.

Background

For those of you familiar with Bash you know that a variable declared inside an if statement or loop can usually be accessed outside that block without issue. unlike other languages (PHP or Java for example) where a variable must be initialized outside the block first. In Bash anything declared inside a block is preserved for the life of the script. Except when that block happens to be a while loop using read from a piped command!

What am I talking about?

Let us look at an example. Feel free to pop these 3 scripts into your console to learn along with us.

Bash script NOT requiring initialized variable

#!/bin/bash
 
#declare nothing for LOOPCOUNT!
 
#but we'll need something for our while loop
i=0
 
while [ $i -lt 5 ]
do
	printf "LOOPCOUNT is %d\n" $LOOPCOUNT
	((LOOPCOUNT++))
 
	((i++))
done
#our count will be preserved
printf "Final LOOPCOUNT is %d\n" $LOOPCOUNT

This will print

eddie@linux-cv2g:~/Scripts> ./LoopDemo.sh
LOOPCOUNT is 0
LOOPCOUNT is 1
LOOPCOUNT is 2
LOOPCOUNT is 3
LOOPCOUNT is 4
Final LOOPCOUNT is 4

As you can see, despite the fact that LOOPCOUNT was first introduced inside the While Loop it is still available after the loop has exited. As I mentioned before this is not true of languages like PHP or Java.

Bash Script ignoring initialized variable !

Now here’s the meat of this article. Despite declaring the variable, and incrementing it properly,  as soon as we leave the loop we lose the value!

This type of while loop is useful for reading each line of a file, or each entry in a directory listing.

#!/bin/bash
 
#ls will list all files in this directory, we pump that through a pipe to While which will act on each entry
 
ls | while read LINE
do
	printf "LOOPCOUNT is %d\n" $LOOPCOUNT
	((LOOPCOUNT++))
 
done
#our count will NOT be preserved
printf "Final LOOPCOUNT is %d\n" $LOOPCOUNT

Since my demo folder has 8 files in it…

This will print

LOOPCOUNT is 0
LOOPCOUNT is 1
LOOPCOUNT is 2
LOOPCOUNT is 3
LOOPCOUNT is 4
LOOPCOUNT is 5
LOOPCOUNT is 6
LOOPCOUNT is 7
Final LOOPCOUNT is 0

Zoot ALors! The variable has amnesia and forgot it’s value!!

Bash Script using While Loop and Read is tamed

#!/bin/bash
 
 
 
#if we break up the ls and while commands we can fix this issue
 
ls > filelist
 
#set stdin to the file listing we jsut made
exec 0<filelist
 
#now act on each line in stdin
while read LINE
do
	printf "LOOPCOUNT is %d\n" $LOOPCOUNT
	((LOOPCOUNT++))
 
done
#our count will once again be preserved
printf "Final LOOPCOUNT is %d\n" $LOOPCOUNT

This will print

LOOPCOUNT is 0
LOOPCOUNT is 1
LOOPCOUNT is 2
LOOPCOUNT is 3
LOOPCOUNT is 4
LOOPCOUNT is 5
LOOPCOUNT is 6
LOOPCOUNT is 7
LOOPCOUNT is 8
Final LOOPCOUNT is 8

(The ls command includes our new script which explains the higher count)

Summary

My best guess is that using a command like

ls | while read LINE; do

opens a sub-shell for the duration of the loop.

This would explain while the value is maintained within the loop but lost immediately after.

I would love to hear from some Bash gurus on this topic!

This applies to log files, mysql backups, or just about any other type of file that is created on a recurring basis. You don’t need a error log from 134 days ago, but error logs for the past week could be very useful. So what do you do? Why recycle of course.

This article shares a simple shell script to purge any files older than X days, where X is of course a number allowing for flexibility. It is very simple to use a shell script to delete log files, or in this example sql backups.

The problem

Your server is being overrun with numerous files that just hang around long after they have served there useful life. These files may be small or large, but something about leaving unused files hanging around doesn’t feel right.

After just a few days of mysql backups I end up with a directory structure like this;

sql_dumps/
|-- edwardawebb.com
|   |-- edwardawebb_wordpress_01-13-2009.sql.gz
|   |-- edwardawebb_wordpress_01-14-2009.sql.gz
|   |-- edwardawebb_wordpress_01-15-2009.sql.gz
|   |-- edwardawebb_wordpress_01-16-2009.sql.gz
|   |-- edwardawebb_wordpress_01-17-2009.sql.gz
|   |-- edwardawebb_wordpress_01-18-2009.sql.gz
|   `-- edwardawebb_wordpress_01-19-2009.sql.gz
|-- mantis.mainsite.org
|   |-- mainsite_mantis_01-13-2009.sql.gz
|   |-- mainsite_mantis_01-14-2009.sql.gz
|   |-- mainsite_mantis_01-15-2009.sql.gz
|   |-- mainsite_mantis_01-16-2009.sql.gz
|   |-- mainsite_mantis_01-17-2009.sql.gz
|   |-- mainsite_mantis_01-18-2009.sql.gz
|   `-- mainsite_mantis_01-19-2009.sql.gz
`-- taskfreak.mainsite.org
    |-- mainsite_taskfreak_01-11-2009
    |-- mainsite_taskfreak_01-11-2009.sql.gz
    |-- mainsite_taskfreak_01-12-2009.sql.gz
    |-- mainsite_taskfreak_01-13-2009.sql.gz
    |-- mainsite_taskfreak_01-14-2009.sql.gz
    |-- mainsite_taskfreak_01-15-2009.sql.gz
    |-- mainsite_taskfreak_01-16-2009.sql.gz
    |-- mainsite_taskfreak_01-17-2009.sql.gz
    |-- mainsite_taskfreak_01-18-2009.sql.gz
    `-- mainsite_taskfreak_01-19-2009.sql.gz

Although 24 files may seem manageable, those who deal with log files and multiple sites know that this can quickly get out of hand.

The solution

We lazily create a shell script to run at weekly intervals to purge all those older files and send them off to the bit bucket.

Only files older than X days should be deleted, we’ll leave all the fresh and potentially needed logs/backups in place

This example assumes mysql logs with the .sql or .sql.gz extensions.

shell script to purge outdated files

#!/bin/bash
 
#if you use this script you must attribute to me Eddie - Edwardawebb.com 1/14/09
 
#this script will run through all nested directories of a parent just killing off all matching files.
 
######
### Set these values
######
 
## default days to retain (override with .RETAIN_RULE in specific directory
DEFRETAIN=60
 
#want to append the activity to a log? good idea, add its location here
LOGFILE=`pwd`/Recycler.log
 
# enter the distinguishing extension, or portion of the filename here (eg. log, txt, etc.)
EXTENSION=sql
 
 
#the absolute path of folder to begin purging
#this is the top most file to begin the attack, all sub directories contain lowercase letters and periods are game.
SQLDIR=$HOME/sql_dumps
 
#####
##   End user configuartion
#####
 
 
#this note will remind you that you have a log in case your getting emails form a cron job or something
echo see $LOGFILE for details
 
#jump to working directory
cd $SQLDIR
 
#if your sub-dirs have some crazy characters you may adjust this regex
DIRS=`ls | grep ^[a-z.]*$`
 
 
TODAY=`date`
 
printf "\n\n********************************************\n\tSQL Recycler Log for:\n\t" | tee -a $LOGFILE
echo $TODAY | tee -a $LOGFILE
printf "********************************************\n" $TODAY | tee -a $LOGFILE
 
for DIR in $DIRS 
do
	pushd $DIR >/dev/null
	HERE=`pwd`
	printf "\n\n%s\n" $HERE | tee -a $LOGFILE
	if [ -f .RETAIN_RULE ]
	then
		printf "\tdefault Retain period being overridden\n" | tee -a $LOGFILE
		read RETAIN < .RETAIN_RULE
	else
		RETAIN=$DEFRETAIN
	fi
 
	printf "\tpurging files older than %s days\n" ${RETAIN} | tee -a $LOGFILE
 
	OLDFILES=`find -mtime +${RETAIN} -regex .*${EXTENSION}.*`
 
	set -- $OLDFILES
 
	if [ -z $1 ]
	then
		printf "\tNo files matching purge criteria\n" | tee -a $LOGFILE
	else
		printf "\tSQL Files being Delete from $HERE\n" | tee -a $LOGFILE
		printf "\t\t%s\n" $OLDFILES  | tee -a $LOGFILE
	fi
 
 	rm -f $OLDFILES
	if [ $? -ne 0 ]
	then	
		echo "Error while deleting last set" | tee -a $LOGFILE
		exit 2
	else
		printf "\tSuccess\n" | tee -a $LOGFILE
	fi
	popd >/dev/null
done

did you notice the bit about .RETAIN_RULE? good!
I added this after I realized that I don’t treat all my sites equally. For this very blog which is backed up daily I only need 3-4 days back max. But for other sites that I back up monthly I need to keep the default 60 days or 1-2 files.

So I set the default in the script to 60. But I allow it to be overwritten by adding a simple text file to any directory. If a file .RETAIN_RULE is present it will read the first line (and first line only!) for a new value, example;

$HOME/sql_dumps/dailysite.com/.RETAIN_RULE

5
#only keep files in this single directory around for 5 days

notice i comment after the actual data!

This means my actual directory structure including retain rules looks more like;

#tree -a sql_dumps
sql_dumps/
|-- edwardawebb.com
|   |-- .RETAIN_RULE
|   |-- edwardawebb_wordpress_01-13-2009.sql.gz
|   |-- edwardawebb_wordpress_01-14-2009.sql.gz
|   |-- edwardawebb_wordpress_01-15-2009.sql.gz
|   |-- edwardawebb_wordpress_01-16-2009.sql.gz
|   |-- edwardawebb_wordpress_01-17-2009.sql.gz
|   |-- edwardawebb_wordpress_01-18-2009.sql.gz
|   `-- edwardawebb_wordpress_01-19-2009.sql.gz
|-- mantis.mainsite.org
|   |-- .RETAIN_RULE
|   |-- mainsite_mantis_01-13-2009.sql.gz
|   |-- mainsite_mantis_01-14-2009.sql.gz
|   |-- mainsite_mantis_01-15-2009.sql.gz
|   |-- mainsite_mantis_01-16-2009.sql.gz
|   |-- mainsite_mantis_01-17-2009.sql.gz
|   |-- mainsite_mantis_01-18-2009.sql.gz
|   `-- mainsite_mantis_01-19-2009.sql.gz
`-- taskfreak.mainsite.org
    |-- mainsite_taskfreak_01-11-2009
    |-- mainsite_taskfreak_01-11-2009.sql.gz
    |-- mainsite_taskfreak_01-12-2009.sql.gz
    |-- mainsite_taskfreak_01-13-2009.sql.gz
    |-- mainsite_taskfreak_01-14-2009.sql.gz
    |-- mainsite_taskfreak_01-15-2009.sql.gz
    |-- mainsite_taskfreak_01-16-2009.sql.gz
    |-- mainsite_taskfreak_01-17-2009.sql.gz
    |-- mainsite_taskfreak_01-18-2009.sql.gz
    `-- mainsite_taskfreak_01-19-2009.sql.gz

The Result

So as the script walks through the structure above it prints a log to the effect of;

see /home/<USERNAME>/sql_dumps/Recycler.log for details
 
 
********************************************
	SQL Recycler Log for:
	Sun Feb 8 00:00:07 PST 2009
********************************************
 
 
/home/MYUSERNAME/sql_dumps/edwardawebb.com
       	default Retain period being overridden
	purging files older than 4 days
	SQL Files being Delete from /home/masterkeedu/sql_dumps/edwardawebb.com
		./edwardawebb_wordpress_01-28-2009.sql.gz
		./edwardawebb_wordpress_02-03-2009.sql.gz
		./edwardawebb_wordpress_01-29-2009.sql.gz
		./edwardawebb_wordpress_02-02-2009.sql.gz
		./edwardawebb_wordpress_01-31-2009.sql.gz
		./edwardawebb_wordpress_01-30-2009.sql.gz
		./edwardawebb_wordpress_02-01-2009.sql.gz
	Success
 
 
/home/MYUSERNAME/sql_dumps/mantis.mainsite.org
	default Retain period being overridden
	purging files older than 4 days
	SQL Files being Delete from /home/masterkeedu/sql_dumps/mantis.mainsite.org
		./webbmaster_mantis_01-30-2009.sql.gz
		./webbmaster_mantis_01-31-2009.sql.gz
		./webbmaster_mantis_02-01-2009.sql.gz
		./webbmaster_mantis_01-27-2009.sql.gz
		./webbmaster_mantis_01-29-2009.sql.gz
		./webbmaster_mantis_02-02-2009.sql.gz
		./webbmaster_mantis_01-28-2009.sql.gz
	Success
 
 
/home/MYUSERNAME/sql_dumps/taskfreak.mainsite.org
        purging files older than 60 days
        No files matching purge criteria
        Success

As with any article I welcome feedback or questions!