Ways to run a shell command

This post is part of a three-part series: #1, #2, #3.

There are several ways to run a shell command.

This post summarizes Chapter 7 of Efficient Linux at the Command Line (2022) by Daniel J. Barrett.

Conditional lists (&&, ||)

Run commands based on success or failure:

cd dir && touch new.txt  # Run 2nd command if 1st succeeds.
cd dir || mkdir          # Run 2nd command only if the first fails.

Combine both:

(cd dir || mkdir dir) && cd dir || echo "I failed"

Exit codes:

• every command returns an exit code
• 0 → success
• non-zero value → failure
• to check the exit code of most recent command: echo $?

Unconditional lists (;)

Run commands in sequence (ignore success/failure):

sleep 5; cp -a ~/important-files /mnt/backup_drive

Command substitution ($(),``)

Command substitution inserts the output of one command into another:

mv $(grep -l "Artist: Kansas" *.txt) kansas
mv result1.txt result2.txt result3.txt kansas  # Result.

Older syntax with backtits:

mv `grep -l "Artist: Kansas" *.txt` kansas

$() is simpler to nest:

echo Today is $(echo $(gdate +%A) | tr a-z A-Z)!

Command substitution runs the command in a subshell.

Process substitution (<())

Process substitution treats command output like a file. It allows commands that read only from disk files (like cat or cp) to read from stdin:

cat <(ls -1 *.jpg | sort -n)

cp <(ls -1 *.jpg | sort -n) ~/Desktop/list.txt

Example:

diff <(ls -1 *.jpg | sort -n) <(seq 1 1000 | sed 's/$/.jpg/')

• you have files 1.jpg–1000.jpg, but some are missing
• to find them, compare existing filenames with the full list using diff
• because diff needs files (not stdin), use process substitution:
  • it runs the command in a subshell
  • and exposes both lists as files to diff

Pass commands to bash (-c)

Pass a command to bash as a string:

bash -c "ls -l"

Useful to run whole commands as the superuser:

• sudo echo "New log file" > /var/log/custom.log fails:
  • after pressing Enter, the shell processes > first
  • it tries to create the file without permission → "Permission denied"
  • sudo never runs (it only applies to echo, not the redirection)

• to solve this, run the whole command as root:

  sudo bash -c 'echo "New log file" > /var/log/custom.log'
  

Pipe commands to bash (| bash)

You can pipe commands directly into bash:

echo "ls -l" | bash

This enables a repeatable workflow for commands: generate → review → execute.

E.g. batch rename files by adding a prefix:

# Check command with `less` first.
ls -1 *.txt | sed 's/.*/mv "&" "old_&"/' | less

# Execute after review.
ls -1 *.txt | sed 's/.*/mv "&" "old_&"/' | bash

Run commands remotely (ssh)

ssh can execute a single command on the remote host (quicker than logging in, running the command and logging out):

ssh myhost.example.com ls

Control output redirection:

ssh myhost.example.com ls > outfile         # Creates outfile on local host.

ssh myhost.example.com "ls > outfile"       # Creates outfile on remote host.

Commands can be piped to ssh to run them on the remote host:

echo "ls > outfile" | ssh myhost.example.com

If you see message about pseudo-terminals or pseudo-ttys, use -T to prevent the SSH server from allocating a terminal:

echo "ls > outfile" | ssh -T myhost.example.com

If you see message like "Welcome to Linux!" then tell ssh explicitely to run bash:

echo "ls > outfile" | ssh myhost.example.com bash

Run a list of commands (xargs)

xargs:

• takes input from stdin as input strings separated by whitespace
• turns it into arguments for a command template

Any command can run recursively by combining find and xargs.

Example, count lines in all Python files:

• pipe the file listing to xargs to serve as its input strings

• and provide wc -l to serve as the command template

  find . -type f -name \*.py -print0 | xargs -0 wc -l
  

Safety with find and xargs:

• xargs -0
  • by default, xargs splits input on whitespace (spaces, tabs, newlines)
  • this becomes a problem when input strings (e.g. filenames) contain whitespace → one filename can be interpreted as multiple arguments
  • so xargs -0 tells xargs to use the null character (ASCII zero) as the separator instead
  • since \0 cannot appear in filenames, this makes it safe
• find -print0
  • separates input strings by null
• note that ls can't do that:
  • but we can convert newlines to nulls with tr
  • ls | tr '\n' '\0' | xargs -0 …

Most important xargs options:

• -n

  • controls how many arguments are appended by xargs onto each generated command

  • the default is to append as many arguments as will fit the shell's limits:

    ls | xargs echo
    ls | xargs -n1 echo    # One argument per echo command.
    ls | xargs -n2 echo    # Two arguments per echo command.
    

• -I

  • controls where the input strings appear in the generated command

  • follow -I whith any string which becomes a placeholder for input strings:

    ls -1 | xargs -I FOO echo FOO rocks
    

  • note: -I limits xargs to one input string per generated command

xargs is a problem solver when commands lines grow very long:

# May fail when evaluating a string > 14 million characters: "Argument list too long".
rm *.txt

# Use `xargs` to split the list of files accross multiple `rm` commands.
find . -maxdepth 1 -name \.txt -type f -print0 | xargs -0 rm

Run a command in the background (&)

Run a command in the background with an ampersand &:

sleep 20 &

• the output may appear (a) at any time or (b) not until the next time we press enter

• the amperstand is also a list operator like && and ||:

  command1 & command2 & command3 &  # All 3 commands in the background.
  

Suspend + resume in the background:

sleep 20
Crtl-z    # Suspend the command.
bg        # Resume the command in the background.

Backgroung commands are part of a shell feature called job control:

• a job is a shell's unit of work with a job ID
• a job = one or more processes (e.g., a pipeline of 6 programs is a single job including 6 processes)

Job control commands:

Screen output from job can appear at any time while the job runs. To avoid clutter, redirect stdout to a file:

sort /usr/share/dict/words | head -n2 > ~/Desktop/results.txt &

Explicit subshells (())

To launch a subshell explicitely, enclose a command in parentheses:

(cd /usr/local && ls)
pwd  # `cd /usr/local` occured in a subshell.

This is useful, for example, when a pipeline needs to change directories mid-execution.

Process replacement (exec)

When we run a command:

• the shell runs it in a separate process which will be destroyed when the command exits
• this behavior can be changed with the builtin exec command

The exec command replaces the current shell process with another command:

• the shell does not create a new process
• instead, it becomes the new command

When the new command exits:

• no shell prompt will follow because the original shell is gone
• if you were in a terminal, it may close
• if it was a login shell, you may be logged out

bash                # Run a child shell.
PS1="Doomed> "      # Change the new shell's prompt.
echo hello          # Run any command you like.
exec ls             # ls replaces the child shell, runs and exits.

Reasons to run exec (mostly in shell scripts):

• save resources by not launching a second process
  • shell scripts use this optimization on the final command of the script
  • if the script runs millions of times, the saving might be worth it

• reassign stdin, stdout or sderr for the current shell

  • this is a second ability of exec

  • exec can redirect stdout for an entire script instead of redirecting the output of each command

    #!/bin/bash
    # Redirect stdout for this script.
    exec > /tmp/outfile
    # All subsequent commands print to /tmp/outfile
    
    
    echo "My name is $USER"
    echo "My current directory is $PWD"