Kubernetes CronJob — lessons from running 50+ in production

Kubernetes CronJob is a thin wrapper that creates a Job at scheduled times. The scheduling is borrowed from robfig/cron (a Go cron library); the resource orchestration is the standard Kubernetes Job controller. That sounds simple. In a production cluster with 50+ scheduled jobs, a few defaults will reliably bite you. This is the list.

concurrencyPolicy: Forbid almost always

The default is Allow. Each scheduled fire creates a new Job regardless of whether the previous one is still running. If a slow nightly export takes 26 hours one night because a downstream DB was slow, the next night's Job starts on top of last night's. Two Jobs hit the same database, both fail, your pager goes off.

spec:
  concurrencyPolicy: Forbid

Forbid means "if the previous Job is still running, skip this scheduled fire". Replace means "kill the previous one and start this one". Allow is the right answer roughly never for a non-trivial workload — pick the policy that matches your actual failure mode.

startingDeadlineSeconds — tune per job cadence

If the controller-manager is down or busy when a scheduled fire was supposed to happen, Kubernetes may backfill missed runs. The control is startingDeadlineSeconds: if a missed schedule is older than this many seconds, it is permanently skipped.

Defaults: there is no default. If you don't set it, the controller will try to start a missed Job "forever", which can mean a cluster recovery from a 4-hour outage starts 240 minute-aligned Jobs simultaneously. That is rarely what you want.

• Per-minute jobs: startingDeadlineSeconds: 60. Skip anything older than 60s.
• Hourly jobs: 3600. Allow one hour of recovery slack.
• Daily jobs: 86400. Recover an entire day if the cluster comes back within 24h.
• Quarterly jobs: 2592000 (30 days). Recover even if the cluster missed a multi-day outage. The cost of missing a quarterly run is high; the cost of running it slightly late is low.

Job history limits — etcd will thank you

Defaults: successfulJobsHistoryLimit: 3, failedJobsHistoryLimit: 1. Completed Jobs are retained as Kubernetes resources (pods linger, the Job object itself lingers) until you replace them with newer ones.

In a cluster with 50 CronJobs at */5 frequency, that is 50 × (3 successful + 1 failed) = 200 retained Job objects at steady state, each with a Pod, each consuming etcd quota and inflating the Kubernetes API response size. Watch this in clusters with many CronJobs:

spec:
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 3   # keep more failures than successes
  ttlSecondsAfterFinished: 3600

ttlSecondsAfterFinished (stable since 1.21) is the cleaner mechanism: finished Jobs are garbage-collected automatically after the TTL. Use it on every CronJob. The default behaviour of retaining failed Pods forever is a slow leak that surfaces as "why is our Kubernetes API slow" six months later.

.spec.timeZone (stable since 1.27)

Before 1.27, all schedules ran in the controller-manager's local timezone — usually UTC. As of 1.27, you can set .spec.timeZone:

spec:
  timeZone: "Europe/Berlin"
  schedule: "0 9 * * 1-5"

This reads the IANA database baked into the controller-manager binary. Make sure your cluster is on at least v1.27; older clusters silently ignore the field. The recommendation if you don't need wall-clock alignment: schedule in UTC anyway, because every downstream timestamp is UTC and you avoid DST entirely.

restartPolicy + backoffLimit — the failure cascade

The Job's Pod has a restartPolicy. For CronJobs, it must be either OnFailure or Never. The interaction with backoffLimit (the Job-level retry count) trips people up:

• restartPolicy: OnFailure — Pod restarts in place after a failure.backoffLimit counts in-Pod restarts. Faster, no scheduling latency between attempts.
• restartPolicy: Never — Pod stays failed; Job controller creates a new Pod for each retry. backoffLimit counts whole-Pod attempts. Slower (scheduling latency per retry) but each attempt gets a clean process.

For idempotent, side-effect-free jobs, OnFailure is faster. For jobs that may have partial state from a previous attempt, Never gives you a clean slate per try. The default backoffLimit: 6 is usually too high — three retries are typically enough; more just delays the alert.

Resource requests — don't skip them

A CronJob without resource requests is the Pod that starves your cluster at 03:00. Set requests and limits explicitly:

resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

A scheduled Job that runs unattended at 03:00 has no human watching it. Without limits, a memory leak quietly eats node memory until something else gets OOM-killed.

The "missed schedule" log line everyone ignores

When the controller-manager skips a fire (because startingDeadlineSeconds elapsed, or because Forbid blocked the start), it logs:

Cannot determine if job needs to be started: too many missed start times (> 100).
Set or decrease .spec.startingDeadlineSeconds or check clock skew.

If you see this in kube-controller-manager logs, the schedule has effectively stopped firing. The fix is to delete and recreate the CronJob, or to reducestartingDeadlineSeconds. Either way, a missed-schedule alarm should fire before you find this in logs.

Monitoring — kube-state-metrics is your friend

Three metrics matter for CronJobs:

• kube_cronjob_status_last_schedule_time — the unix timestamp of the last time the CronJob fired. Compare against expected cadence to detect missed fires.
• kube_job_status_failed — count of failed Jobs in the recent history window. Filter by the CronJob label.
• kube_job_status_succeeded — count of successful Jobs. Track the ratio against _failed as a health signal.

A simple Prometheus alert: job has not fired in 2× the expected cadence. For an hourly job, alert if time() - kube_cronjob_status_last_schedule_time > 7200.

The image-pull cliff

A CronJob at */5 frequency that always uses an image tagged latest with imagePullPolicy: Always hits the registry 288 times a day. If your registry is rate-limited (Docker Hub anonymous users: 100 pulls per 6 hours per IP), the schedule starts silently failing the moment you cross the limit. Use immutable image tags (digest or semver) and set imagePullPolicy: IfNotPresent for high-frequency jobs.