TopolearnOdd serialization errors, slow pipelines and controller overload all trace back to how pipelines execute.
Pipeline code runs on the controller, not the agent. Only steps like sh run on agents. To survive controller restarts, the Groovy is transformed into continuation-passing style (CPS): after each step the program state (local variables, call stack) is serialized to disk, and a resumed build continues from there.
Consequences:
- Variables held across steps must be Serializable. Objects like
java.util.regex.Matcheror iterators causeNotSerializableException. - Methods annotated
@NonCPSrun as normal Groovy, faster and without serialization, but they cannot call pipeline steps and are not resumable. - Heavy Groovy logic (parsing large files, big loops) runs on the controller's CPU and blocks other builds. Do such work in
shwith a real program on the agent.
Durability settings trade safety for speed: MAX_SURVIVABILITY writes state often; PERFORMANCE_OPTIMIZED writes much less, so a crash may lose a running build's progress, but I/O drops drastically. Set it globally or with options { durabilityHint('PERFORMANCE_OPTIMIZED') }.
Scaling rules from the Jenkins docs: keep pipelines thin, use few steps (one sh with a script beats 200 tiny sh calls), avoid large stashes and huge console logs, and give the controller zero executors.
Example
// Problem: Matcher isn't Serializable and lives across a step
// def m = (version =~ /(\d+)\.(\d+)/); sh 'make'; echo m[0][1] -> NotSerializableException
@NonCPS
def parseMajor(String version) {
def m = version =~ /(\d+)\.(\d+)/
return m ? m[0][1] : null // plain Groovy, no steps allowed here
}
pipeline {
agent any
options { durabilityHint('PERFORMANCE_OPTIMIZED') }
stages {
stage('Version') {
steps {
script {
def major = parseMajor(readFile('VERSION').trim())
echo "Major version ${major}" // Major version 3
}
sh './ci/build-and-test.sh' // one step, real work on the agent
}
}
}
}Remember
- Pipeline Groovy runs on the controller as serialized CPS code
- @NonCPS methods are plain Groovy and cannot call steps
- PERFORMANCE_OPTIMIZED durability trades resumability for much less I/O
Watch out for
- Doing heavy parsing in pipeline Groovy, loading the controller instead of an agent.