Performance: forks, pipelining, async

Ansible · Expert · 12 min

A playbook that takes an hour on 500 hosts can often take minutes with the right settings.

Where time goes: for each task and host, Ansible opens an SSH session, creates a temp directory, copies the module, runs it and cleans up. Optimisations attack each part:

  • forks (default 5): how many hosts run a task in parallel. Raise to 25-100 for big fleets, limited by control-node CPU and memory.
  • SSH multiplexing: ControlMaster/ControlPersist (on by default in Ansible's ssh args) reuses one TCP+auth connection for many tasks.
  • pipelining (pipelining = True): sends the module over the existing session's stdin instead of copying a file, cutting several SSH operations per task. Requires that sudo doesn't need requiretty.
  • Facts: gather_facts: false when not needed, gather_subset to collect less, or fact caching (jsonfile, redis) with gathering = smart to reuse facts between runs.
  • Strategies: the default linear waits for all hosts at each task; free lets each host run ahead independently. serial batches trade speed for safety.
  • async: async: 600, poll: 0 starts a long task (a big download, a backup) in the background and moves on; check it later with async_status.
  • Fewer tasks: pass lists to package modules instead of looping, and use template once rather than many lineinfile edits.

Measure before tuning: the profile_tasks and timer callbacks (callbacks_enabled = ansible.posix.profile_tasks, ansible.posix.timer) show per-task timings.

Example

# ansible.cfg
[defaults]
forks = 50
gathering = smart
fact_caching = jsonfile
fact_caching_connection = .facts_cache
fact_caching_timeout = 86400
callbacks_enabled = ansible.posix.profile_tasks, ansible.posix.timer

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=300s

# playbook snippet:
#   - hosts: all
#     strategy: free
#     gather_facts: false
#     tasks:
#       - ansible.builtin.command: /usr/local/bin/backup.sh
#         async: 3600
#         poll: 0
#         register: backup_job

Remember

  • Raise forks; keep SSH multiplexing; enable pipelining
  • Skip, subset or cache facts
  • free strategy and async for independent or long tasks; profile_tasks to measure

Watch out for

  • Enabling pipelining on hosts whose sudoers require a TTY, making become tasks fail.

Check yourself

  1. What does pipelining save?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: docs.ansible.com/ansible/latest/playbook_guide/playbooks_strategies.html