TopolearnRunning independent tasks concurrently can turn a ten-minute script into a one-minute one.
Ending a command with & starts it in the background: the shell continues immediately and stores the job's PID in $!. wait blocks until all background children finish; wait "$pid" waits for one and returns its exit status, which is how you detect failures of parallel tasks. Bash 5.1 added wait -n -p var to wait for whichever job finishes next and report its PID.
In an interactive shell, job control manages jobs: jobs lists them, Ctrl-Z suspends the foreground job, bg resumes it in the background and fg %1 brings job 1 back. Background jobs die with the terminal unless started with nohup or detached with disown; for anything long-running, prefer a proper service manager, tmux or screen.
For many small tasks, do not start thousands of & jobs at once. Limit concurrency:
xargs -P 4 -n 1 cmdruns up to four commands at a time from input lines (use-0withfind -print0for safe file names),- GNU
paralleloffers more control, - or count running jobs and call
wait -nbefore starting another.
Output from parallel jobs interleaves unpredictably; write each job's output to its own file, or prefix lines. Remember each background job runs in a subshell, so it cannot set variables in the parent.
Example
sleep 0.2 & pid1=$!
(sleep 0.1; exit 3) & pid2=$!
wait "$pid2"; echo "job 2 status: $?" # job 2 status: 3
wait "$pid1"; echo "job 1 status: $?" # job 1 status: 0
work() { sleep "0.$(( RANDOM % 3 ))"; echo "done $1"; }
for item in a b c; do work "$item" & done
wait # all three done, order varies
printf '%s\n' 1 2 3 4 5 6 |
xargs -P 3 -I{} sh -c 'sleep 0.1; echo "item {}"' # 3 at a time
max=2
for n in 1 2 3 4; do
while (( $(jobs -rp | wc -l) >= max )); do wait -n; done
work "job$n" &
done
waitRemember
- cmd & runs in the background; $! is its PID
- wait "$pid" returns that job's exit status
- Cap concurrency with xargs -P or wait -n
Watch out for
- Starting jobs with & and never calling wait, so the script exits before they finish or their failures go unnoticed.