TopolearnStage-by-stage pipelines wait for the slowest job; needs can cut total pipeline time dramatically.
With stages alone, a job waits for every job in all earlier stages. needs: changes that: a job with needs: [build-api] starts as soon as build-api succeeds, even if other jobs of earlier stages are still running. The pipeline becomes a directed acyclic graph (DAG), visible in the pipeline view as dependency lines.
Details that matter:
- A job with
needs: []starts immediately when the pipeline is created. - With
needs, a job downloads artifacts only from the jobs it needs. Useneeds: [{job: build, artifacts: false}]to wait without downloading. - If a needed job is not in the pipeline (because its rules excluded it), the pipeline fails to be created, unless the need is marked
optional: true. - A job can need up to 50 jobs by default, and can even need jobs in the same stage.
A common use is a monorepo pipeline where frontend and backend each build, test and deploy independently: each chain progresses at its own speed instead of all frontend tests waiting for the slowest backend build.
Stages still matter as a visual grouping and for jobs without needs, so most pipelines combine the two.
Example
stages: [build, test, deploy]
build-api: {stage: build, script: [make api], artifacts: {paths: [api/bin/]}}
build-web: {stage: build, script: [make web], artifacts: {paths: [web/dist/]}}
test-api:
stage: test
needs: [build-api] # starts when build-api is done, not build-web
script: [make test-api]
lint:
stage: test
needs: [] # starts immediately
script: [make lint]
deploy-web:
stage: deploy
needs:
- job: build-web # gets only web/dist/
- job: test-api
artifacts: false
script: [./deploy-web.sh]Remember
- needs: lets a job start as soon as its dependencies finish
- needs: [] starts a job immediately
- With needs, artifacts come only from the needed jobs
Watch out for
- Needing a job that rules removed from this pipeline; mark it optional: true.