Environment, working directory and PATH

subprocess · Practical · 9 min

Build and deploy scripts need to run tools in a specific folder with specific variables set.

By default a child inherits your current working directory and a copy of your environment.

cwd= runs the command in another directory without changing your own process's directory. It accepts a str or a pathlib.Path, and relative paths in the command are resolved against it.

env= replaces the child's environment entirely. Usually you want to add to it:

env = {**os.environ, "NODE_ENV": "production"}
subprocess.run(["npm", "run", "build"], env=env, cwd=project)

Passing only {"NODE_ENV": "production"} drops PATH, HOME and everything else, which breaks most programs. All keys and values must be strings.

Finding programs: the executable is searched on PATH (the child's PATH from env on POSIX). shutil.which("ffmpeg") returns the full path or None, letting you fail early with a helpful message. For Python itself, prefer sys.executable to make sure the child uses the same interpreter and virtual environment as the parent.

To run a module with the same interpreter, use [sys.executable, "-m", "pip", "install", ...], the reliable way to call pip from code.

Example

import os, shutil, subprocess, sys, tempfile
from pathlib import Path

tmp = Path(tempfile.mkdtemp())
(tmp / "data.txt").write_text("hi")

r = subprocess.run(["ls"], cwd=tmp, capture_output=True, text=True)
print(r.stdout.strip())                      # data.txt

env = {**os.environ, "GREETING": "hello"}
code = "import os; print(os.environ['GREETING'], 'PATH' in os.environ)"
r = subprocess.run([sys.executable, "-c", code], env=env, capture_output=True, text=True)
print(r.stdout.strip())                      # hello True

print(shutil.which("sh"))                    # /usr/bin/sh (or /bin/sh)
print(shutil.which("no-such-tool"))          # None

Remember

  • cwd= sets the child's working directory only
  • env= replaces the environment: merge with os.environ
  • shutil.which finds executables; sys.executable is the current Python

Watch out for

  • Passing env={'KEY': 'v'} alone, which removes PATH and breaks the command.

Check yourself

  1. subprocess.run(['make'], env={'DEBUG': '1'}) fails with FileNotFoundError or odd errors. Why?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: docs.python.org/3/library/subprocess.html#popen-constructor