Parallel builds with -j

Make · Advanced · 12 min

Parallel builds cut build times by the number of cores, but only if the dependency graph is honest.

make -j8 runs up to eight recipes at once; make -j$(nproc) uses every core. make only starts a target once all its prerequisites are done, so a correct dependency graph is automatically parallel-safe.

Parallel builds expose missing dependencies. If app needs gen/config.h but the rule doesn't say so, a serial build may work by luck of ordering while -j builds fail randomly. Fix the graph rather than adding .NOTPARALLEL.

Tools for a precise graph:

  • Order-only prerequisites, after a |: build/%.o: src/%.c | build means "build must exist first, but its timestamp doesn't matter". Perfect for output directories, whose timestamps change whenever a file is added to them.
  • A grouped target a.c a.h &: spec.y (GNU make 4.3+) says one recipe produces several files together. A plain a.c a.h: spec.y means two independent rules, and under -j the generator may run twice at once.
  • --output-sync=target (-O) buffers each target's output so logs from parallel jobs don't interleave.

When make calls itself recursively, use $(MAKE), so the jobserver shares the job slots instead of each sub-make starting its own eight. MAKEFLAGS += -j8 inside the Makefile is possible but surprises users; usually leave -j to the caller.

Example

SRC := $(wildcard src/*.c)
OBJ := $(SRC:src/%.c=build/%.o)

build/app: $(OBJ)
	cc -o $@ $^

# order-only: the directory must exist, its timestamp is ignored
build/%.o: src/%.c gen/version.h | build
	cc -Igen -c $< -o $@

build:
	mkdir -p $@

# grouped target: one run of the generator produces both files
gen/version.h gen/version.c &: version.txt
	@mkdir -p gen
	./mkversion.sh version.txt gen/version.h gen/version.c

# $ make -j"$(nproc)" -O

Remember

  • make -jN runs independent recipes in parallel
  • Random -j failures mean the dependency graph is missing an edge
  • Use | for order-only prerequisites (directories) and &: for grouped targets

Watch out for

  • Making object files depend normally on their output directory, so every new file in it triggers rebuilds.

Check yourself

  1. Why put an output directory after | in a prerequisite list?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: www.gnu.org/software/make/manual/make.html#Parallel