Kubernetes CronJobs schedule containerized workloads — backups, data pipelines, report generation, cache warming, cleanup tasks. Like all cron systems, Kubernetes has a fundamental gap: it schedules jobs but doesn't tell you when they fail. A CronJob that crashes, times out, or quietly stops running produces no automatic notification unless you build one.
This guide covers what Kubernetes CronJob failures look like, how to detect them with native tooling, and how to add the monitoring needed to catch a missed or failed job before your users do.
The Kubernetes scheduler fires a CronJob on schedule by creating a Job object. The Job creates a Pod. The Pod runs your container. If the container exits with a non-zero code or the Pod fails to start, Kubernetes records the failure — but it does not send an alert. The next scheduled run creates a new Job and Pod, regardless of whether the previous one succeeded.
This means the default failure mode for a Kubernetes CronJob is: the job stops producing correct output, and nothing notifies you.
Common scenarios:
Your container throws an unhandled exception and exits with code 1. Kubernetes marks the Job as failed, retries according to backoffLimit, eventually stops retrying. The CronJob keeps scheduling new runs. Each run fails. Nobody knows.
Kubernetes has an activeDeadlineSeconds field on Jobs. If your job exceeds it, the Job is terminated and marked failed. The partial work it completed may look like success until someone compares timestamps.
If you set concurrencyPolicy: Forbid and a previous run is still active, Kubernetes skips the new run entirely — no error, no log, just a skipped execution. With startingDeadlineSeconds misconfigured, runs can be missed due to controller downtime without any visible trace.
Kubernetes CronJob schedules use standard five-field cron syntax in UTC. A job scheduled at 0 9 * * 1-5 runs at 9:00 AM UTC, which is 1:00 AM Pacific. If the author intended 9:00 AM Pacific, the job runs at the wrong time. It runs successfully — just not when expected. This is an intent mismatch, not a failure, but it has the same downstream effect.
Start with what's already there:
# List all Jobs created by a CronJob
kubectl get jobs -n my-namespace --selector=cronjob-name=my-cronjob
# Or use label selectors Kubernetes creates automatically
kubectl get jobs -n my-namespace -l job-nameLook at the COMPLETIONS column — a 0/1 that doesn't flip to 1/1 within the expected window is a failed job.
kubectl describe job/my-cronjob-28760400 -n my-namespaceThe Events section will show BackoffLimitExceeded if retries were exhausted, or DeadlineExceeded if the job timed out.
# Find the Pod from a failed Job
kubectl get pods -n my-namespace --selector=job-name=my-cronjob-28760400
# Read its logs
kubectl logs my-cronjob-28760400-xxxxx -n my-namespace
# If the Pod already terminated
kubectl logs my-cronjob-28760400-xxxxx -n my-namespace --previouskubectl describe cronjob my-cronjob -n my-namespaceThe Events section will show MissSchedule events if Kubernetes skipped runs due to controller downtime or ConcurrencyPolicy.
Checking manually is fine for debugging. It's not monitoring. Real monitoring means you receive an alert within minutes of a failure — not hours after a user complaint.
If you're already running Prometheus in your cluster (common with kube-prometheus-stack), you can add a CronJob failure alert rule:
groups:
- name: cronjob.rules
rules:
- alert: KubeCronJobFailed
expr: |
kube_job_status_failed > 0
and
kube_job_owner{owner_kind="CronJob"} == 1
for: 5m
labels:
severity: warning
annotations:
summary: "CronJob {{ $labels.job_name }} failed"
description: "Job {{ $labels.job_name }} in namespace {{ $labels.namespace }} has failed."
- alert: KubeCronJobNotScheduled
expr: |
time() - kube_cronjob_next_schedule_time > 3600
for: 5m
labels:
severity: critical
annotations:
summary: "CronJob {{ $labels.cronjob }} hasn't run in over an hour"Route these through Alertmanager to Slack, PagerDuty, or email. This approach gives you granular control but requires a running Prometheus stack.
The most reliable pattern for CronJob monitoring is a dead man's switch: the job pings an external service at the end of each successful run. If the ping doesn't arrive within the expected window, the external service alerts you.
Add a final step to your CronJob container that fires a heartbeat on success:
command: ["/bin/sh", "-c"]
args:
- |
python /app/my_job.py && \
curl -s "https://cron.watch/ping/YOUR_JOB_TOKEN" --retry 3If the job crashes before reaching the curl, the heartbeat never fires, and the monitor alerts you. This approach is simple and catches both job crashes and missed schedules — even when the scheduler itself fails.
The key advantage over Prometheus: it works even if your cluster monitoring stack goes down. The heartbeat either arrives or it doesn't.
For teams that want alerting without an external dependency, you can add an initContainer that registers a "job started" event, and a completion hook via lifecycle.postStart or a wrapper script that sends a "job ended" event on both success and failure to an internal service.
This is the most complex option and requires maintaining the alerting sidecar. Most teams find the dead man's switch approach simpler and equally reliable.
These settings affect observability and should be set on every production CronJob:
apiVersion: batch/v1
kind: CronJob
metadata:
name: my-cronjob
spec:
schedule: "0 9 * * 1-5" # UTC — verify this matches your intent
concurrencyPolicy: Forbid # Prevent overlapping runs
startingDeadlineSeconds: 300 # Allow 5 min window after scheduled time
successfulJobsHistoryLimit: 3 # Keep 3 recent successful job records
failedJobsHistoryLimit: 5 # Keep 5 recent failed job records (for debugging)
jobTemplate:
spec:
backoffLimit: 2 # Retry up to 2 times before marking failed
activeDeadlineSeconds: 3600 # Kill the job after 1 hour (adjust per job)
template:
spec:
restartPolicy: Never # Don't restart the container — retry via Job
containers:
- name: my-job
image: my-image:latest
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"Important settings:
startingDeadlineSeconds: if the controller is down when a job is scheduled, Kubernetes counts missed schedules. More than 100 missed schedules causes the CronJob to stop scheduling. Setting this to a short window (5–15 min) prevents cascading misses.failedJobsHistoryLimit: higher values preserve debugging history. Lower values reduce cluster overhead. 3–5 is a reasonable default.restartPolicy: Never: for Jobs, Never means Kubernetes creates a new Pod for each retry instead of restarting in place. This preserves logs from each attempt.| Approach | Setup complexity | Catches missed schedules | Works if cluster is down | Cost |
|---|---|---|---|---|
| kubectl manual checks | None | ✓ (manual) | ✗ | Free |
| Prometheus alerts | High | Partial | ✗ | Free (infra cost) |
| Dead man's switch | Low (one curl) | ✓ | ✓ | Free–$9/mo |
| Datadog / New Relic | Medium | ✓ | ✓ | $$$ (enterprise pricing) |
For small to medium teams running a handful of CronJobs, the dead man's switch pattern is typically the best tradeoff: low setup cost, reliable, and works independently of the Kubernetes cluster's health.
The container image can't be pulled — wrong tag, registry auth expired, private registry unreachable. The Pod status shows ImagePullBackOff. Check kubectl describe pod for the specific error. The job never starts.
The container exceeded its memory limit and was killed by the kernel. The exit code is 137. Increase resources.limits.memory or fix the memory leak. The job ends abruptly, often producing partial output that looks like success.
A ConfigMap or Secret the job depends on was deleted or renamed. The Pod status shows CreateContainerConfigError. The job never starts.
The job's ServiceAccount doesn't have permission to access the Kubernetes API resources it needs. The container starts but fails on the first API call. Logs show permission denied errors.
Kubernetes CronJob schedules are always in UTC. A developer in Pacific time writing 0 9 * * * expects 9 AM local — it runs at 2 AM local. The job runs, but at the wrong time. Use a cron expression parser to confirm what UTC time your schedule actually fires.
Add one line to your CronJob container command. CronWatch sends an alert if the ping doesn't arrive within the expected window — catching crashes, timeouts, missed schedules, and scheduler downtime in a single check.
command: ["/bin/sh", "-c"]
args: ["python job.py && curl -s https://cron.watch/ping/TOKEN"]Kubernetes CronJobs fail silently by default. The three most important steps to add monitoring:
failedJobsHistoryLimit to 3–5 so failed jobs are visible when you look.curl line. This catches crashes, timeouts, and missed schedules without requiring Prometheus.The Prometheus alerting approach is more powerful if you already run kube-prometheus-stack, but for most small teams the dead man's switch pattern is faster to set up and more reliable.