Skip to content
Share this..

Scope Issue in Bash While Loops

2009 February 7
tags: bash, read, while
by Eddie

I was writing a pretty complex Log Recylcer script that handles rotation, purging and compression.  I wanted to print a nice summary at the end with all the counts.  I got incredibly frustrated when a simple variable increment seemed to have scope issue. I can only re-create the issue using a pipe and read.

Background

For those of you familiar with Bash you know that a variable declared inside an if statement or loop can usually be accessed outside that block without issue. unlike other languages (PHP or Java for example) where a variable must be initialized outside the block first. In Bash anything declared inside a block is preserved for the life of the script. Except when that block happens to be a while loop using read from a piped command!

What am I talking about?

Let us look at an example. Feel free to pop these 3 scripts into your console to learn along with us.

Bash script NOT requiring initialized variable

#!/bin/bash
 
#declare nothing for LOOPCOUNT!
 
#but we'll need something for our while loop
i=0
 
while [ $i -lt 5 ]
do
	printf "LOOPCOUNT is %d\n" $LOOPCOUNT
	((LOOPCOUNT++))
 
	((i++))
done
#our count will be preserved
printf "Final LOOPCOUNT is %d\n" $LOOPCOUNT

This will print

eddie@linux-cv2g:~/Scripts> ./LoopDemo.sh
LOOPCOUNT is 0
LOOPCOUNT is 1
LOOPCOUNT is 2
LOOPCOUNT is 3
LOOPCOUNT is 4
Final LOOPCOUNT is 4

As you can see, despite the fact that LOOPCOUNT was first introduced inside the While Loop it is still available after the loop has exited. As I mentioned before this is not true of languages like PHP or Java.

Bash Script ignoring initialized variable !

Now here’s the meat of this article. Despite declaring the variable, and incrementing it properly,  as soon as we leave the loop we lose the value!

This type of while loop is useful for reading each line of a file, or each entry in a directory listing.

#!/bin/bash
 
#ls will list all files in this directory, we pump that through a pipe to While which will act on each entry
 
ls | while read LINE
do
	printf "LOOPCOUNT is %d\n" $LOOPCOUNT
	((LOOPCOUNT++))
 
done
#our count will NOT be preserved
printf "Final LOOPCOUNT is %d\n" $LOOPCOUNT

Since my demo folder has 8 files in it…

This will print

LOOPCOUNT is 0
LOOPCOUNT is 1
LOOPCOUNT is 2
LOOPCOUNT is 3
LOOPCOUNT is 4
LOOPCOUNT is 5
LOOPCOUNT is 6
LOOPCOUNT is 7
Final LOOPCOUNT is 0

Zoot ALors! The variable has amnesia and forgot it’s value!!

Bash Script using While Loop and Read is tamed

#!/bin/bash
 
 
 
#if we break up the ls and while commands we can fix this issue
 
ls > filelist
 
#set stdin to the file listing we jsut made
exec 0<filelist
 
#now act on each line in stdin
while read LINE
do
	printf "LOOPCOUNT is %d\n" $LOOPCOUNT
	((LOOPCOUNT++))
 
done
#our count will once again be preserved
printf "Final LOOPCOUNT is %d\n" $LOOPCOUNT

This will print

LOOPCOUNT is 0
LOOPCOUNT is 1
LOOPCOUNT is 2
LOOPCOUNT is 3
LOOPCOUNT is 4
LOOPCOUNT is 5
LOOPCOUNT is 6
LOOPCOUNT is 7
LOOPCOUNT is 8
Final LOOPCOUNT is 8

(The ls command includes our new script which explains the higher count)

Summary


My best guess
is that using a command like

ls | while read LINE; do

opens a sub-shell for the duration of the loop.

This would explain while the value is maintained within the loop but lost immediately after.

I would love to hear from some Bash gurus on this topic!

8 Responses leave one →
  1. guns permalink
    April 25, 2009

    Hi Eddie, came in from google (and your site is great).

    > My best guess is that using a command like
    > $ ls | while read LINE; do
    > opens a sub-shell for the duration of the loop.

    You’re close. Reading stdin from a pipe always opens a subshell, which you seem to recognize has its own local variable scope. This is most often apparent in piping text into a while loop.

    > #if we break up the ls and while commands we can fix this issue
    > $ ls > filelist
    > #set stdin to the file listing we jsut made
    > $ exec 0

        $ while read LINE; do
        >     ((++LOOPCOUNT))
        >     echo "$LOOPCOUNT: $LINE"
        > done < <(ls -l)

    ‘< ( command )' returns a file descriptor, usually at /dev/fd/*. The output of your command thus becomes immediately available as a file for use as stdin of any command or loop.

    If you don't have file descriptors at your disposal, or want to write strictly POSIX-compliant scripts, store your stdout in a temporary variable rather than a temporary file. You can then read it into your while loop as a file using a here string:

        > STDOUT=$(ls -l)
        > while read LINE; do
        >     ((++LOOPCOUNT))
        >     echo "$LOOPCOUNT: $LINE"
        > done <<< "$STDOUT"

    < << "String" presents the string as a file and is a handy replacement for the usual echo "foo bar" | grep "bar" scenario:

    $ grep "bar" <<< "foo bar"

    Hope that helps. I’m just happy to see that other people realize that the bash shell is worthy of some attention.

  2. Eddie permalink*
    April 25, 2009

    @guns

    Thanks! I would be a liar if I said I never learned from my readers, and its always a treat.

    I was also thinking that named pipes (which are visible to the filesystem) would be a good solution.
    http://www.linuxjournal.com/content/using-named-pipes-fifos-bash

  3. Aaron permalink
    April 29, 2009

    Dudes! Guns and Eddie, this post rocks! Thanks so much. This tendency of bash to start a sub-shell when piping can be a nuisance. Now I have finally found a consistent way around it. Thanks again!

  4. Chris permalink
    January 22, 2010

    Great post! This helped to unstick me from this annoying quirk of bash. I’m using bash from cygwin, and it did not consider “<<<" to be valid. I was able to work around it as follows:

    STDOUT=$(ls -l)
    while read LINE; do
    ((++LOOPCOUNT))
    echo "$LOOPCOUNT: $LINE"
    done <<EOF
    $STDOUT
    EOF

    Of course this assumes that "EOF" does not occur in your input stream. Probably not a valid assumption for doing directory listings, but works fine when you know your input does not contain your special token like "EOF"

  5. Bill permalink
    February 12, 2010

    I’d really like to express my thanks for the advice offered here!

  6. T.J. permalink
    August 20, 2010

    Thanks guys, between the OP and guns I was able to get rid of a bug in a script I was working on. Basically I was reading from a file in a while loop, but if the last line also contained EOF it wouldn’t get passed into the loop. Also I was having the variable scoping issues. Basically the solution was to cat the file into a variable, then use guns <<< "string" trick to send the contents into the read while loop. When that happens, the last line gets ran through the loop, irregardless whether or not the last line is also the end of the file.

    sFileContents=$( cat file.txt )
    
    while read var1 var2
    do
      #it all works!
    done <<< "${sFileContents}"
    
  7. Jakub permalink
    August 25, 2011

    Thanks for great post and solution. Solution from here saved my day. :)

  8. marleo permalink
    December 4, 2011

    ((++THANKYOU)) – Thought I was going mad.

Leave a Reply

Note: You can use basic XHTML in your comments. Your email address will never be published.

Subscribe to this comment feed via RSS