Scope Issue in Bash While Loops
I was writing a pretty complex Log Recylcer script that handles rotation, purging and compression. I wanted to print a nice summary at the end with all the counts. I got incredibly frustrated when a simple variable increment seemed to have scope issue. I can only re-create the issue using a pipe and read.
Background
For those of you familiar with Bash you know that a variable declared inside an if statement or loop can usually be accessed outside that block without issue. unlike other languages (PHP or Java for example) where a variable must be initialized outside the block first. In Bash anything declared inside a block is preserved for the life of the script. Except when that block happens to be a while loop using read from a piped command!
What am I talking about?
Let us look at an example. Feel free to pop these 3 scripts into your console to learn along with us.
Bash script NOT requiring initialized variable
#!/bin/bash #declare nothing for LOOPCOUNT! #but we'll need something for our while loop i=0 while [ $i -lt 5 ] do printf "LOOPCOUNT is %d\n" $LOOPCOUNT ((LOOPCOUNT++)) ((i++)) done #our count will be preserved printf "Final LOOPCOUNT is %d\n" $LOOPCOUNT
This will print
eddie@linux-cv2g:~/Scripts> ./LoopDemo.sh LOOPCOUNT is 0 LOOPCOUNT is 1 LOOPCOUNT is 2 LOOPCOUNT is 3 LOOPCOUNT is 4 Final LOOPCOUNT is 4
As you can see, despite the fact that LOOPCOUNT was first introduced inside the While Loop it is still available after the loop has exited. As I mentioned before this is not true of languages like PHP or Java.
Bash Script ignoring initialized variable !
Now here’s the meat of this article. Despite declaring the variable, and incrementing it properly, as soon as we leave the loop we lose the value!
This type of while loop is useful for reading each line of a file, or each entry in a directory listing.
#!/bin/bash #ls will list all files in this directory, we pump that through a pipe to While which will act on each entry ls | while read LINE do printf "LOOPCOUNT is %d\n" $LOOPCOUNT ((LOOPCOUNT++)) done #our count will NOT be preserved printf "Final LOOPCOUNT is %d\n" $LOOPCOUNT
Since my demo folder has 8 files in it…
This will print
LOOPCOUNT is 0 LOOPCOUNT is 1 LOOPCOUNT is 2 LOOPCOUNT is 3 LOOPCOUNT is 4 LOOPCOUNT is 5 LOOPCOUNT is 6 LOOPCOUNT is 7 Final LOOPCOUNT is 0
Zoot ALors! The variable has amnesia and forgot it’s value!!
Bash Script using While Loop and Read is tamed
#!/bin/bash #if we break up the ls and while commands we can fix this issue ls > filelist #set stdin to the file listing we jsut made exec 0<filelist #now act on each line in stdin while read LINE do printf "LOOPCOUNT is %d\n" $LOOPCOUNT ((LOOPCOUNT++)) done #our count will once again be preserved printf "Final LOOPCOUNT is %d\n" $LOOPCOUNT
This will print
LOOPCOUNT is 0 LOOPCOUNT is 1 LOOPCOUNT is 2 LOOPCOUNT is 3 LOOPCOUNT is 4 LOOPCOUNT is 5 LOOPCOUNT is 6 LOOPCOUNT is 7 LOOPCOUNT is 8 Final LOOPCOUNT is 8
(The ls command includes our new script which explains the higher count)
Summary
My best guess is that using a command like
ls | while read LINE; do
opens a sub-shell for the duration of the loop.
This would explain while the value is maintained within the loop but lost immediately after.
I would love to hear from some Bash gurus on this topic!
Hi Eddie, came in from google (and your site is great).
> My best guess is that using a command like
> $ ls | while read LINE; do
> opens a sub-shell for the duration of the loop.
You’re close. Reading stdin from a pipe always opens a subshell, which you seem to recognize has its own local variable scope. This is most often apparent in piping text into a while loop.
> #if we break up the ls and while commands we can fix this issue
> $ ls > filelist
> #set stdin to the file listing we jsut made
> $ exec 0
‘< ( command )' returns a file descriptor, usually at /dev/fd/*. The output of your command thus becomes immediately available as a file for use as stdin of any command or loop.
If you don't have file descriptors at your disposal, or want to write strictly POSIX-compliant scripts, store your stdout in a temporary variable rather than a temporary file. You can then read it into your while loop as a file using a here string:
< << "String" presents the string as a file and is a handy replacement for the usual echo "foo bar" | grep "bar" scenario:
Hope that helps. I’m just happy to see that other people realize that the bash shell is worthy of some attention.
@guns
Thanks! I would be a liar if I said I never learned from my readers, and its always a treat.
I was also thinking that named pipes (which are visible to the filesystem) would be a good solution.
http://www.linuxjournal.com/content/using-named-pipes-fifos-bash
Dudes! Guns and Eddie, this post rocks! Thanks so much. This tendency of bash to start a sub-shell when piping can be a nuisance. Now I have finally found a consistent way around it. Thanks again!
Great post! This helped to unstick me from this annoying quirk of bash. I’m using bash from cygwin, and it did not consider “<<<" to be valid. I was able to work around it as follows:
STDOUT=$(ls -l)
while read LINE; do
((++LOOPCOUNT))
echo "$LOOPCOUNT: $LINE"
done <<EOF
$STDOUT
EOF
Of course this assumes that "EOF" does not occur in your input stream. Probably not a valid assumption for doing directory listings, but works fine when you know your input does not contain your special token like "EOF"
I’d really like to express my thanks for the advice offered here!
Thanks guys, between the OP and guns I was able to get rid of a bug in a script I was working on. Basically I was reading from a file in a while loop, but if the last line also contained EOF it wouldn’t get passed into the loop. Also I was having the variable scoping issues. Basically the solution was to
catthe file into a variable, then use guns<<< "string"trick to send the contents into the read while loop. When that happens, the last line gets ran through the loop, irregardless whether or not the last line is also the end of the file.sFileContents=$( cat file.txt ) while read var1 var2 do #it all works! done <<< "${sFileContents}"Thanks for great post and solution. Solution from here saved my day.
((++THANKYOU)) – Thought I was going mad.