Unit Types#

systemd manages the entire system through “units,” each representing a resource or service. The most common types:

  • service: Daemons and long-running processes (nginx, postgresql, your application).
  • timer: Scheduled execution, replacing cron. More flexible and better integrated with logging.
  • socket: Network sockets that trigger service activation on connection. Enables lazy startup and zero-downtime restarts.
  • target: Groups of units that represent system states (multi-user.target, graphical.target). Analogous to SysV runlevels.
  • mount: Filesystem mount points managed by systemd.
  • path: Watches filesystem paths and activates units when changes occur.

Unit files live in three locations, in order of precedence: /etc/systemd/system/ (local admin overrides), /run/systemd/system/ (runtime, non-persistent), and /usr/lib/systemd/system/ (package-installed defaults). Always put custom units in /etc/systemd/system/.

Creating a Service Unit#

A service unit file has three sections:

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application Server
Documentation=https://docs.example.com/myapp
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
Environment=NODE_ENV=production
EnvironmentFile=/opt/myapp/.env
ExecStartPre=/opt/myapp/bin/check-config
ExecStart=/opt/myapp/bin/server --port 8080
ExecStop=/bin/kill -SIGTERM $MAINPID
Restart=on-failure
RestartSec=5
StartLimitBurst=5
StartLimitIntervalSec=60
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

[Unit] section: After controls startup ordering (start after these units). Wants declares a weak dependency (start these if available, but do not fail if they are missing). Requires declares a hard dependency (fail if this unit is not available).

[Service] section: ExecStart is the main process command. ExecStartPre runs before the main process (validation, migrations). User and Group set the process identity. WorkingDirectory sets the current directory. Environment and EnvironmentFile set environment variables.

[Install] section: WantedBy determines which target pulls this service in when enabled. multi-user.target means the service starts during normal multi-user boot.

Restart Policies#

Restart behavior is critical for production services. The policy determines what happens when the process exits:

  • Restart=on-failure: Restart only on non-zero exit codes, signals, and timeouts. The most common choice for production services – it restarts on crashes but not on clean shutdowns (exit code 0).
  • Restart=always: Restart regardless of exit status. Use for services that should never stop (container runtimes, critical infrastructure).
  • Restart=on-abnormal: Restart on signals, timeouts, and watchdog events, but not on clean or unclean exit codes. Useful when the service might exit intentionally with a non-zero code.
Restart=on-failure
RestartSec=5                    # wait 5 seconds between restarts
StartLimitBurst=5               # max 5 restarts...
StartLimitIntervalSec=60        # ...within 60 seconds

The StartLimitBurst and StartLimitIntervalSec parameters prevent restart loops. Without them, a service with a configuration error will restart indefinitely, consuming resources and flooding logs. When the limit is reached, systemd stops trying and marks the unit as failed. You must systemctl reset-failed myapp before you can start it again.

Service Types#

The Type directive tells systemd how to determine when the service is “ready”:

  • simple (default): systemd considers the service started immediately after ExecStart runs. The process must not fork. This is correct for most modern applications.
  • forking: The process forks into the background (traditional Unix daemon style). systemd waits for the parent to exit and tracks the child via PIDFile. Use this for legacy daemons.
  • oneshot: The process runs to completion and exits. systemd waits for it to finish before starting dependent units. Good for initialization scripts.
  • notify: The process sends a notification to systemd when it is ready (sd_notify()). This is the most accurate type – systemd knows exactly when the service is serving traffic. Applications must be instrumented to send the notification.
# For a forking daemon
Type=forking
PIDFile=/run/myapp/myapp.pid

# For a oneshot initialization
Type=oneshot
RemainAfterExit=yes              # consider the unit "active" even after the process exits

Resource Controls#

systemd uses Linux cgroups to enforce resource limits on services. These limits are hard-enforced by the kernel:

[Service]
# Memory limit -- OOM killer activates if exceeded
MemoryMax=2G
MemoryHigh=1.5G                  # soft limit, kernel starts reclaiming aggressively

# CPU quota -- percentage of one CPU core
CPUQuota=200%                    # allow up to 2 full cores

# Maximum number of tasks (threads + processes)
TasksMax=512

# I/O weight (relative priority, 1-10000, default 100)
IOWeight=200

# Protect the service from OOM killer (use with caution)
OOMScoreAdjust=-500

MemoryMax is a hard ceiling. When a process exceeds it, the kernel’s OOM killer terminates it. MemoryHigh is a soft limit – the kernel aggressively reclaims memory from the cgroup but does not kill the process. Using both together provides a buffer zone.

TasksMax prevents fork bombs and runaway thread creation. The default is 4915 (33% of the kernel limit), which is usually sufficient but may need increasing for applications that use many threads.

Timer Units#

Timers replace cron with better logging, dependency management, and calendar expressions. A timer activates a corresponding service unit (same name with .service extension).

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily Database Backup

[Timer]
OnCalendar=*-*-* 03:00:00       # daily at 3:00 AM
Persistent=true                   # run missed timers after reboot
RandomizedDelaySec=600            # add up to 10 min random delay (avoid thundering herd)
AccuracySec=1min                  # wake up within 1 minute of scheduled time

[Install]
WantedBy=timers.target
# /etc/systemd/system/backup.service
[Unit]
Description=Database Backup

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh
User=backup

Calendar expressions are flexible:

OnCalendar=*-*-* 03:00:00         # daily at 3 AM
OnCalendar=Mon *-*-* 09:00:00     # every Monday at 9 AM
OnCalendar=*-*-01 00:00:00        # first of every month at midnight
OnCalendar=hourly                  # shorthand for *-*-* *:00:00

Boot-relative timers are also available:

OnBootSec=5min                    # 5 minutes after boot
OnUnitActiveSec=1h                # 1 hour after the service last ran

Persistent=true is important for backup and maintenance timers. If the system was powered off when the timer should have fired, systemd runs it immediately on next boot.

Manage timers with:

systemctl list-timers --all       # show all timers and their next fire time
systemctl start backup.timer      # start the timer
systemctl enable backup.timer     # enable on boot

Journal: Centralized Logging#

systemd’s journal captures stdout/stderr from all services, kernel messages, and syslog messages in a structured, indexed binary format.

journalctl -u myapp               # all logs for a service
journalctl -u myapp -f            # follow in real time
journalctl -u myapp --since "1 hour ago"   # time-scoped
journalctl -u myapp -p err        # errors and above (emerg, alert, crit, err)
journalctl -u myapp -n 50         # last 50 lines
journalctl -u myapp --no-pager -o json-pretty  # structured JSON output

Journal Persistence#

By default, the journal may store logs only in memory (/run/log/journal/), which means logs are lost on reboot. To persist:

mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal

Or configure in /etc/systemd/journald.conf:

[Journal]
Storage=persistent          # auto (default), persistent, volatile, none
SystemMaxUse=2G             # max disk usage for journal
SystemKeepFree=1G           # keep at least this much disk free
MaxRetentionSec=30day       # delete entries older than 30 days

Restart the journal: systemctl restart systemd-journald.

Socket Activation#

Socket activation lets systemd listen on a socket and start the service only when a connection arrives. Benefits include: zero-downtime restarts (systemd holds the socket while the service restarts, queuing connections), lazy startup (services that rarely receive traffic start on-demand), and parallel boot (services do not wait for each other to bind ports).

# /etc/systemd/system/myapp.socket
[Unit]
Description=My Application Socket

[Socket]
ListenStream=8080
Accept=no                        # hand the socket to the service (not per-connection)

[Install]
WantedBy=sockets.target

The service must be written to accept the socket file descriptor from systemd (file descriptor 3, or use the sd_listen_fds API).

Common Operations Reference#

systemctl start myapp             # start now
systemctl stop myapp              # stop now
systemctl restart myapp           # stop then start
systemctl reload myapp            # send SIGHUP (if supported)
systemctl status myapp            # show status, PID, recent logs
systemctl enable myapp            # start on boot
systemctl disable myapp           # do not start on boot
systemctl enable --now myapp      # enable and start immediately
systemctl daemon-reload           # reload unit file changes
systemctl list-units --failed     # show failed units
systemctl reset-failed myapp      # clear failed state

Debugging#

systemctl status myapp            # first stop -- shows exit code, recent logs
journalctl -u myapp -xe           # recent entries with explanations
systemd-analyze blame             # boot time per service (find slow starters)
systemd-analyze critical-chain    # boot dependency chain
systemd-analyze verify myapp.service  # check unit file syntax

Common gotcha: forgetting daemon-reload. After editing any unit file in /etc/systemd/system/, you must run systemctl daemon-reload. Without it, systemd uses the cached version of the unit file. This is the single most common systemd mistake.

Common gotcha: ExecStart must be an absolute path. ExecStart=myapp fails. It must be ExecStart=/usr/local/bin/myapp. systemd does not search PATH. This also applies to ExecStartPre, ExecStop, and all other Exec directives.