This week I ran into a confusing issue when trying to use grep in a shell script that was called via a Makefile. In hindsight, the error should have been obvious, but in the moment I was lost. So, I hope this simple description of my problem and the solution helps someone else in the future.

I had a simple text file that contained a list of files that existed in a certain directory. That file looked something like:

/tmp/instructions.txt
/tmp/results_a.csv
/tmp/results_b.csv
/tmp/observations.txt
/tmp/results_c.csv

The existing shell script looped through several directories and created a single temporary file that listed every file in the directories that it examined. A change needed to be made to exclude all of the .txt files that were found. So, I figured I could do something like:

cat list_of_files.txt | grep -v ".txt"

The -v modifier for grep inverts the search. So, we would grab each line of the file that does not contain the string ".txt". Perfect!

This logic existed in a shell script and there was a Makefile with a make command that would run this script. That didn't seem to be a problem. I updated the shell script with this new logic, saved the file, and ran the make command. And things seemed to work great...until make start reporting the following error:

make: *** [parse-files] Error 1

This error was telling me that the parse-files command that was defined in my Makefile was exiting with an exit code of 1. parse-files is the make command that went and ran the shell script to build and parse the list of files.

I couldn't figure out why the introduction of the grep command was causing an exit code of 1. Until I did some looking around about grep's possible exit codes. When grep doesn't find any lines to match on, it sets the exit code to 1. So, consider this list of files:

/tmp/instructions.txt
/tmp/observations.txt

The special case here is that I don't have any .csv files in my list. So, what happens what grep runs on this file with my given command? Well, see for yourself.

% cat files.txt | grep -v ".txt"
% echo $?
1

grep returns nothing - which makes sense as there were no lines that did not contain ".txt" - while also setting the exit code to 1.

This is what was causing make command to error out. make will complain anytime that exit code is not equal to 0. In my particular case though, I don't want it to error out if no files matched. That simply means no .csv files were found in that directory, so I can simply move onto the next file. I do still want to catch other errors that make complains about (i.e. syntax errors). What can I do?

After some more looking around for ideas online, I decided on updating the logic in the shell script to this:

cat files.txt | ( grep -v ".txt" || [[ $? == 1 ]] )

If grep doesn't find any matches the exit code will be set to 1, but since I am now also checking if the exit code ($?) is 1, then that will result to true and keep the exit code at 0. Assuming the rest of the script executes without issue, this will mean that the make command will succeed as the exit code will be 0.

In hindsight, this should have been obvious. But, it wasn't. The make error wasn't all that helpful and this script was running on a production environment where I felt like it could have been several other issues before simply looking at the grep command.

P.S. - This blog post was extremely helpful as I was doing some research and debugging!