TopolearnCron jobs and CI tasks finish before any scrape can happen, yet 'last backup succeeded?' is a vital alert.
The Pushgateway (port 9091) is a small cache: jobs POST or PUT metrics to it, and Prometheus scrapes the gateway like any target. It exists for one case: service-level batch jobs that have no instance to scrape, such as a nightly backup or a database migration.
The URL encodes a grouping key: /metrics/job/<job>/instance/<instance>. PUT replaces every metric in that group, POST replaces only metrics with the same names, DELETE removes the group. Pushed metrics carry the grouping labels, so the Prometheus job must set honor_labels: true or job becomes exported_job.
The gateway adds push_time_seconds and push_failure_time_seconds per group; alert on time() - push_time_seconds > 86400 to catch a job that stopped running.
Why it is not for everything:
- Metrics never expire. A decommissioned job keeps reporting its last value forever until someone deletes it, and
uptells you only whether the gateway is reachable, not the job. - It is a single point of failure and bottleneck, with no aggregation: two instances pushing the same series overwrite each other.
- Pushing from long-running services throws away the pull model's health checking and discovery.
Alternatives: for host-local cron jobs use node_exporter's textfile collector; for many short tasks inside a service, count them in the service's own metrics; for OpenTelemetry push, use the 3.x OTLP receiver.
Client libraries offer push_to_gateway helpers, and curl --data-binary works for shell scripts.
Example
# Push from a backup script (text format on stdin)
cat <<EOF | curl --data-binary @- http://pushgateway:9091/metrics/job/db_backup/instance/db1
# TYPE backup_last_success_timestamp_seconds gauge
backup_last_success_timestamp_seconds $(date +%s)
# TYPE backup_size_bytes gauge
backup_size_bytes 73400320
EOF
# Delete a group when a job is retired
curl -X DELETE http://pushgateway:9091/metrics/job/db_backup/instance/db1
# prometheus.yml
# - job_name: pushgateway
# honor_labels: true
# static_configs: [{targets: ["pushgateway:9091"]}]
# Alert: backup older than a day
# time() - backup_last_success_timestamp_seconds{job="db_backup"} > 86400Remember
- Only for service-level batch jobs with nothing to scrape
- Grouping key in the URL; honor_labels: true on the scrape job
- Pushed metrics persist until deleted; alert on push_time_seconds age
Watch out for
- Using the Pushgateway to turn Prometheus into a push system for regular services, losing up and discovery.