Rewriting history at scale

Git · Expert · 15 min

Leaked credentials and bloated repos need whole-history rewrites, which are easy to get wrong.

Interactive rebase edits a handful of commits. To change every commit, for example to remove a leaked key, purge a 2 GB video, or extract a subdirectory into its own repository, you need a filtering tool.

git filter-branch exists but is slow and full of pitfalls; its own documentation recommends git filter-repo, a separately installed Python tool (pip install git-filter-repo) that is fast and safe by default. Common operations:

  • --path secrets.env --invert-paths: delete a path from all history.
  • --strip-blobs-bigger-than 50M: drop huge files.
  • --replace-text expressions.txt: replace strings, such as API keys, in every version of every file.
  • --path src/lib/ --path-rename src/lib/:: keep one directory and make it the root.
  • --mailmap: fix author names and emails.

Every rewritten commit gets a new ID, so this is a coordinated migration, not a quick fix:

  • Work in a fresh clone (filter-repo insists on it, and removes the origin remote so you can't push by accident).
  • Rotate leaked secrets first. Clones, forks, CI caches and PR refs on the host may still hold old commits; rewriting doesn't un-leak anything.
  • Announce a freeze, force-push all branches and tags (git push --force --all and --tags), and have everyone re-clone. An old clone merged back in reintroduces the purged history.
  • Ask the host to purge cached refs where needed.

Alternatives: git replace grafts history without rewriting (for example to stitch an archived history onto a fresh start), and for big binaries going forward, Git LFS keeps them out of the object store.

Example

pip install git-filter-repo
git clone --mirror git@github.com:acme/app.git app-clean
cd app-clean

# One pass: drop a file, huge blobs, and redact key-like strings
printf 'regex:AKIA[0-9A-Z]{16}==>REDACTED\n' > ../expr.txt
git filter-repo --path config/prod.env --invert-paths \
  --strip-blobs-bigger-than 50M --replace-text ../expr.txt
git log --all --oneline -S AKIA        # verify: no output

git remote add origin git@github.com:acme/app.git
git push --force --mirror origin      # after secrets are rotated and team is warned

Remember

  • Use git filter-repo, not filter-branch, for whole-history rewrites
  • Rotate leaked secrets: rewriting history doesn't un-leak them
  • Every commit ID changes: freeze, force-push everything, everyone re-clones

Watch out for

  • Letting a teammate push from an old clone after the rewrite, which brings the purged objects back.

Check yourself

  1. You purged a leaked API key from history and force-pushed. What else is mandatory?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: git-scm.com/docs/git-filter-branch

Builds on

Unlocks

Nothing builds on this yet.