← Blog  ·  October 2026

How to Monitor systemd Timers: Alert When Scheduled Jobs Fail

systemd timers are better than cron — until one silently stops firing and nobody notices for three weeks.

systemd timers have largely replaced crontab on modern Linux systems. They're more powerful: you can set precise activation conditions, control the execution environment, log everything through journald, and track last-run state with systemctl status. But they share cron's most dangerous property — when a timer stops firing, nothing tells you by default.

This guide covers how systemd timers work, how to inspect their status, common failure modes, and how to add external monitoring so you're alerted before a user notices the scheduled job hasn't run.

How systemd timers work

A systemd timer consists of two unit files working together:

• myservice.timer — defines the schedule (when to run)
• myservice.service — defines the job (what to run)

A minimal timer that runs a backup script every night at 2am:

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily database backup

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

# /etc/systemd/system/backup.service
[Unit]
Description=Database backup job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

Enable and start the timer (not the service — systemd activates the service on schedule):

systemctl enable backup.timer
systemctl start backup.timer

The Persistent=true option is important: if the system was off when the job should have run, it will run on the next boot. Without it, missed runs are silently skipped.

Checking timer status manually

These are the commands you'll use when investigating a timer:

List all active timers

systemctl list-timers

Output includes NEXT (next activation time), LAST (last activation time), PASSED (time since last run), and UNIT. Look at the PASSED column — if a daily job shows 3 days, it hasn't run.

Check whether a timer and its service are healthy

systemctl status backup.timer
systemctl status backup.service

Look for Active: active (waiting) on the timer. On the service, check the last exit code under the Main PID line — a non-zero exit means the job failed.

View the service execution log

journalctl -u backup.service --since "24 hours ago"

This shows all stdout/stderr from the last 24 hours. If the job threw an exception or hit a permission error, it's here.

Check last activation time programmatically

systemctl show backup.timer --property=LastTriggerUSec

Returns a microsecond timestamp. Compare against the current time to calculate how long ago the timer last fired.

Common systemd timer failure modes

1. Timer enabled but not started

Enabling a timer (systemctl enable) creates the symlink to auto-start it on boot, but doesn't start it in the current session. If you enabled the timer but forgot systemctl start, it won't run until the next reboot.

# Check — should show "Active: active (waiting)"
systemctl is-active backup.timer

2. Service exits non-zero but timer keeps scheduling

The timer is happy as long as it successfully activated the service. It doesn't care whether the service's ExecStart command succeeded or failed. A backup script that fails every night shows Active: active (waiting) on the timer while silently failing every run.

Check the service's last result:

systemctl show backup.service --property=ExecMainStatus

A value of 0 is success. Anything else is a failure that the timer is not reporting.

3. Transient service failures masked by auto-restart

If your .service unit has Restart=on-failure, a failed run will be retried. The second attempt might succeed, making it look healthy in the status output — while the underlying issue (a database connection timing out, a rate limit being hit) fires on every first attempt.

4. The server or VM was rebuilt

After a server rebuild, provisioning automation re-creates the service file but forgets to run systemctl enable backup.timer && systemctl start backup.timer. The scheduled job stops running. Nobody notices for days or weeks because the last successful run was before the rebuild and no alert was configured.

5. OnCalendar expression is wrong

systemd calendar expressions are not the same as cron expressions. A common mistake:

# This fires every weekday (correct)
OnCalendar=Mon..Fri *-*-* 09:00:00

# This fires at 09:00 on any day — includes weekends
OnCalendar=*-*-* 09:00:00

Test your expression before deploying:

systemd-analyze calendar "Mon..Fri *-*-* 09:00:00"

Adding monitoring to systemd timers

Manual inspection only tells you the current state. For production timers, you want to be alerted when a job doesn't run on schedule — without manually SSHing into a server to check.

Option 1: OnFailure= directive

systemd can invoke a separate service unit when any service fails. Create a notification service:

# /etc/systemd/system/notify-failure@.service
[Unit]
Description=Send failure notification for %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/send-alert.sh "systemd service %i failed"

Then add to your timer's service unit:

[Unit]
OnFailure=notify-failure@backup.service

This catches failures, but not the case where the timer stops firing entirely — if the timer itself is inactive, OnFailure never triggers.

Option 2: Heartbeat monitoring (catches both failures and missed runs)

The more reliable pattern is a heartbeat: your service sends a signal at the end of a successful run. External monitoring detects the absence of that signal and alerts you.

Add a heartbeat ping to the end of your backup.service:

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=/usr/bin/curl -s "https://your-monitor.com/api/heartbeat/YOUR_TOKEN" > /dev/null

The monitor expects a ping every N minutes (your timer interval + some grace period). If the ping doesn't arrive, it alerts you. This approach catches:

• The service running but failing (no ping sent)
• The timer stopped firing entirely (no ping sent)
• The server being down (no ping sent)
• The service running too slowly (ping arrives late)

The ExecStartPost= directive only runs when ExecStart= exits 0, so a failed backup won't send a false-positive ping.

Option 3: Prometheus blackbox exporter

If you're already running Prometheus, the blackbox exporter can probe HTTP endpoints. Combine it with a /health/last-backup endpoint in your application that returns a timestamp. Write an alerting rule in Alertmanager:

- alert: BackupStale
  expr: time() - backup_last_success_timestamp_seconds > 86400 + 900
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Database backup hasn't run in over 24h"

This works well if you already have Prometheus infrastructure. If you don't, it's significant overhead for one timer.

A shell script to audit all timers

Run this on any server to get a quick health report of all active timers and whether they've missed their expected window:

#!/bin/bash
# systemd-timer-audit.sh — report timers that haven't fired recently

echo "=== systemd timer health report ==="
echo ""

systemctl list-timers --no-legend --no-pager | while read -r next_time next_left last_time last_ago unit rest; do
  # Skip timers with no last-run time
  if [[ "$last_time" == "n/a" ]]; then
    echo "⚠  $unit — has never fired"
    continue
  fi
  echo "✓  $unit — last fired $last_ago ago, next in $next_left"
done

echo ""
echo "=== Failed service units ==="
systemctl list-units --state=failed --type=service --no-legend 2>/dev/null | awk '{print "✗ " $1}'

This gives you a quick picture across all timers. Integrate it into a daily report email or a monitoring script.

Summary

Manual checks work for debugging. For production timers that need to run reliably, add an external heartbeat monitor so you're alerted before users notice the job has stopped.

← More articles