← Back to Posts

For All the Files (Shells 225 excerpt)

Iterating over files in Bash with 'for'.

Your fingers fly over the keyboard as you work in your terminal window. That command is done, so you hit ↑ once, then Ctrl+← until you get to the right of the filename, Ctrl+W to delete the entire word, scroll up to find the next file from your earlier ls, then copy/paste or type its name with tab-completion, and finally hit Enter. You’re so good at Bash… right?

You’re not and here’s why. This is an excerpt from my presentation “Shells 225: Bash for Security Professionals” that I presented at NolaSec on .

If that sounded like part of your work day, you could work smarter. You’re missing out on one of the simplest means of automation possible—the Bash for loop. Bash has many control structures, but one of the most flexible and useful of these is for.

Bash for can be used to iterate over half a dozen different types of items, from static lists to ranges to arrays to the contents of variables. But one of the most useful and flexible of these is files.

for filename in process-*.csv process-*.tsv ; do
  mycommand "${filename}" >> output.json
done

Now, in reality, this would be written by you on the command line without the extra newlines, as below.

for filename in process-*.csv process-*.tsv ; do mycommand "${filename}" >> output.json; done

But in the interests of readability, I’m going to show the formatted versions.

Uses of Bash for with files include

  • Applying Yara checks in multiple files over the same memory capture
  • Applying a single Yara check over multiple memory captures
  • Applying Yara checks in multiple files over multiple memory captures (nested for)
  • Executing the same command with different configuration files specified on the command line
  • Processing every file in a directory

So we get patterns like:

for i in *.yara; do
  mycommand "${i}" capture.lime
done

And the pattern:

for i in *; do
  [[ ! -f "${i}" ]] && continue
  mycommand -i "${i}" -transform -o "${i%.*}.out"
done

This iterates over all files in the directory (but no subdirectories) that don’t start with a dot. And because the list of files is always gathered when for is invoked, you can output files into the same directory without being worried they’ll feed back into the input. Add some parameter expansion magic (${i%.*}.out) and you don’t need to fiddle with output filenames, either.

The real benefit here is that you can very easily turn these for commands into shell scripts to automate your work. Automation is repeatable, which provides benefits

  • more consistent work
  • less rework
  • documenting institutional knowledge

But even more so, it’s something you can commit to a git repository and share, leading to more efficiency across the team and the org.