Managing Services with systemd: systemctl, Unit Overrides, Timers and Cron

Version note: The commands in this post were run on a test VM with Ubuntu Server 24.04.5 LTS (systemd 255, cron 3.0, Nginx 1.24); the code was built with .NET SDK 10.0.402 and Microsoft.Extensions.Hosting.Systemd 10.0.15 and run on the ASP.NET Core 10.0.15 runtime.

Guess First

You wrote a systemd unit file for the API of Bookstore, the online bookstore used as the shared example across this site's series, put it on the server and started it. Everything looks fine:

$ sudo systemctl daemon-reload
$ sudo systemctl start bookstore-api
$ systemctl is-active bookstore-api
active
$ curl -s http://127.0.0.1:5000/health; echo
{"status":"Healthy"}

During the night the server restarts for a kernel update. Is Bookstore.Api running in the morning? The unit file also has a Restart=always line. The answer is in the next section, after a real reboot.

In the fourth post we saw that a process started with nohup does not come back after a reboot and does not restart when it crashes. In this post we hand Bookstore over to systemd: day-to-day management with systemctl, where unit files live, drop-in overrides that leave packaged units untouched, dependencies, and a systemd timer and cron for the nightly backup. What each line of the unit file means and the hardening settings are in the systemd post of the Web Servers and Hosting series; here we adapted that unit to the paths of this series and focus on managing it.

The Unit File and Day-to-Day Management with systemctl

We use Type=notify so that the application can tell systemd "I am ready"; for that we added the Microsoft.Extensions.Hosting.Systemd package and a single line to Bookstore.Api:

builder.Services.AddSystemd();

The unit file lives in the repository at deploy/linux/systemd/bookstore-api.service:

[Unit]
Description=Bookstore API (ASP.NET Core on Kestrel)
After=network.target

[Service]
Type=notify
User=bookstore
Group=bookstore
WorkingDirectory=/opt/bookstore/current
ExecStart=/usr/bin/dotnet /opt/bookstore/current/Bookstore.Api.dll
Restart=always
RestartSec=10
TimeoutStopSec=30
SyslogIdentifier=bookstore-api
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=ASPNETCORE_URLS=http://127.0.0.1:5000
Environment=DOTNET_NOLOGO=true
EnvironmentFile=/etc/bookstore/secrets.env

[Install]
WantedBy=multi-user.target

The bookstore user from the third post, the current symlink from the second post and the secrets.env with mode 640 come together here. Because systemd reads the EnvironmentFile as root and passes the variables to the process, the application itself does not need to be able to read the file. To see the state, use status:

$ sudo systemctl status bookstore-api --no-pager -n 4
● bookstore-api.service - Bookstore API (ASP.NET Core on Kestrel)
     Loaded: loaded (/etc/systemd/system/bookstore-api.service; enabled; preset: enabled)
     Active: active (running) since Sat 2026-11-14 15:10:31 UTC; 10s ago
   Main PID: 1140 (dotnet)
      Tasks: 19 (limit: 1106)
     Memory: 53.1M (peak: 53.3M)
        CPU: 3.756s
     CGroup: /system.slice/bookstore-api.service
             └─1140 /usr/bin/dotnet /opt/bookstore/current/Bookstore.Api.dll

Nov 14 15:10:31 bookstore-dev bookstore-api[1140]: Microsoft.Hosting.Lifetime[14] Now listening on: http://127.0.0.1:5000
Nov 14 15:10:31 bookstore-dev bookstore-api[1140]: Microsoft.Hosting.Lifetime[0] Application started. Hosting environment: Production; Content root path: /opt/bookstore/releases/2026-11-14-4
Nov 14 15:10:31 bookstore-dev systemd[1]: Started bookstore-api.service - Bookstore API (ASP.NET Core on Kestrel).
Nov 14 15:10:31 bookstore-dev bookstore-api[1140]: Bookstore.Api[0] Bookstore.Api started, PID 1140

How to read it: the Loaded line shows where the unit file comes from and whether it is enabled, Active the current state and since when, CGroup all processes that belong to the service, and the lines at the bottom the latest journal entries. The log lines do not show without sudo; to read the journal the user must be in the adm or systemd-journal group (sixth post). The commands you use every day:

• start, stop, restart: start the service, stop it (SIGTERM first, SIGKILL when the timeout runs out), stop and start it.
• reload: make the process reread its configuration without stopping it; works only if the unit has ExecReload=.
• enable, disable, enable --now: start at boot or not; --now also starts it right away.
• is-active, is-enabled, is-failed: one-word answers for scripts and health checks.
• systemctl --failed: every unit that has crashed; one of the first things to check when you log in to a server.
• daemon-reload: reread unit files from disk; needed every time you change one.

I saw the reload difference on two services:

$ sudo systemctl reload bookstore-api
Failed to reload bookstore-api.service: Job type reload is not applicable for unit bookstore-api.service.
$ sudo systemctl reload nginx; echo "exit=$?"
exit=0
$ sudo -u deploy sudo -n /usr/bin/systemctl restart bookstore-api; echo "exit=$?"
exit=0

Nginx's unit has an ExecReload: it runs nginx -s reload, which sends SIGHUP to the master process; Bookstore's does not, because, as we saw in the fourth post, .NET shuts down on SIGHUP. In .NET you use restart for configuration changes. The last line shows that the sudoers rule from the third post works: the deploy user can restart only this service.

enable and After a Reboot

The answer to the guess. I started the unit with start and rebooted the VM:

$ uptime
 15:10:24 up 1 min,  1 user,  load average: 5.27, 1.82, 0.65
$ systemctl status bookstore-api --no-pager
○ bookstore-api.service - Bookstore API (ASP.NET Core on Kestrel)
     Loaded: loaded (/etc/systemd/system/bookstore-api.service; disabled; preset: enabled)
     Active: inactive (dead)
$ systemctl is-enabled bookstore-api
disabled
$ sudo systemctl enable --now bookstore-api
Created symlink /etc/systemd/system/multi-user.target.wants/bookstore-api.service → /etc/systemd/system/bookstore-api.service.

The application did not come back. start only means "start now"; to start at boot you need enable. Restart=always only kicks in when the process of a running service dies; it has nothing to do with reboots. What enable does is visible in the output: because of WantedBy=multi-user.target in the [Install] section, it created a symlink under multi-user.target.wants/. At boot, when systemd reaches the normal multi-user system (multi-user.target), it also starts everything in that folder. preset: enabled in the status line is the distribution's default suggestion; when you put the file there yourself, nothing is enabled automatically. On the next reboot the service came up on its own.

Where Unit Files Live and Drop-in Overrides

$ systemctl show -p FragmentPath -p DropInPaths nginx bookstore-api
FragmentPath=/usr/lib/systemd/system/nginx.service
DropInPaths=

FragmentPath=/etc/systemd/system/bookstore-api.service
DropInPaths=

Units that come with packages live in /usr/lib/systemd/system/ (the /lib/systemd/system in older sources is the same place; as we saw in the second post, /lib is now a symlink). This VM has 401 files there, and they are all overwritten when the package is updated. The administrator's units and changes live in /etc/systemd/system/, and that location takes precedence. Bookstore's unit is ours, so we put it there directly.

When you want to change a package's unit, do not edit the file in /usr/lib; the next apt upgrade wipes your change. Use a drop-in instead: a small file that contains only the lines you want to change. Nginx's unit does not restart on a crash; I first saw that with an experiment, then added a drop-in:

$ sudo kill -9 $(systemctl show -p MainPID --value nginx)
$ systemctl is-active nginx
failed
$ sudo systemctl start nginx
$ sudo systemctl edit nginx
Successfully installed edited file '/etc/systemd/system/nginx.service.d/override.conf'.
$ systemctl cat nginx --no-pager | tail -5

# /etc/systemd/system/nginx.service.d/override.conf
[Service]
Restart=on-failure
RestartSec=5s

systemctl edit opens an editor; the part between two comment lines in the file becomes the content of the drop-in, and I wrote three lines there. On save, systemd wrote the file to /etc/systemd/system/nginx.service.d/override.conf and ran daemon-reload itself. systemctl cat shows the package's file followed by every drop-in in order; it is the shortest way to see the settings a service really runs with. I repeated the same crash:

$ sudo kill -9 $(systemctl show -p MainPID --value nginx)
$ systemctl is-active nginx; systemctl show -p NRestarts nginx
active
NRestarts=1
$ sudo journalctl -u nginx --no-pager -n 6 -o short | cut -c1-160
Nov 14 15:13:19 bookstore-dev systemd[1]: nginx.service: Killing process 2343 (nginx) with signal SIGKILL.
Nov 14 15:13:19 bookstore-dev systemd[1]: nginx.service: Killing process 2344 (nginx) with signal SIGKILL.
Nov 14 15:13:19 bookstore-dev systemd[1]: nginx.service: Failed with result 'signal'.
Nov 14 15:13:25 bookstore-dev systemd[1]: nginx.service: Scheduled restart job, restart counter is at 1.
Nov 14 15:13:25 bookstore-dev systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server...
Nov 14 15:13:25 bookstore-dev systemd[1]: Started nginx.service - A high performance web server and a reverse proxy server.

This time systemd waited 5 seconds and restarted Nginx. Setting a value again in a drop-in replaces the previous one; but for lines that can take several values, such as ExecStart=, you first need an empty ExecStart= to clear the list, otherwise systemd thinks you are adding a second command and refuses to start the service; when I tried it, the journal said Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services. Refusing. To undo a drop-in, use sudo systemctl revert nginx.

Dependencies: After, Wants and Requires

Relations between units answer two separate questions: order and dependency. After= is only order: "if both are being started, start me after it". On its own it does not start the other unit. Wants= is a weak dependency: "start it when starting me, and if it fails to start I carry on". Requires= is a strong dependency: if it is explicitly stopped or restarted, I am stopped or restarted too; and if it is combined with After=, I am not started when it fails to start. Wants or Requires is usually written together with After.

$ systemctl show -p After -p Wants -p Requires -p WantedBy bookstore-api
Requires=system.slice sysinit.target -.mount
Wants=
WantedBy=multi-user.target
After=sysinit.target -.mount systemd-journald.socket system.slice basic.target network.target

The only thing we wrote is After=network.target; systemd added the rest by default. network.target only says that network management is up, not that an IP address is ready. That is often the reason an application that connects to a remote database at boot fails to connect and crashes. A unit that really waits for the network to be ready writes Wants=network-online.target and After=network-online.target together, as we do in the backup service. The most robust fix is for the application itself to retry on connection errors; the External Service Integration and Resilience series covers that.

systemd-analyze shows what waits for what during boot and how long it takes. The answer to "why did the service come up late after the reboot" is usually here:

$ systemd-analyze blame --no-pager | head -6
21.648s fwupd-refresh.service
20.276s dev-vda1.device
18.812s cloud-config.service
17.134s snapd.seeded.service
15.931s snapd.service
10.879s udisks2.service
$ systemd-analyze critical-chain bookstore-api.service --no-pager
bookstore-api.service +2.418s
└─basic.target @52.560s
  └─sockets.target @52.557s
    └─snapd.socket @52.488s +65ms
      └─sysinit.target @52.326s
        └─cloud-init.service @44.695s +7.566s
          └─systemd-networkd-wait-online.service @42.922s +1.625s
...

blame shows how long each unit spent starting, critical-chain the chain of waits on the path to a unit: @ is when the unit became ready, + how long it took to start. The times are long on my test VM; on a normal VM they are in the range of seconds, but the way to read them is the same.

A Nightly Backup with a systemd Timer

We want a backup of Bookstore's database every night. The backup script itself belongs to the Bash Scripting series; here /usr/local/bin/bookstore-db-backup (under deploy/linux/bin/ in the repository) is a placeholder that prints a single line for now, and we only care about when and how it runs. A timer consists of two files: a service that does the work and a timer that triggers it.

# /etc/systemd/system/bookstore-db-backup.service
[Unit]
Description=Nightly BookstoreDb backup
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=bookstore
Group=bookstore
ExecStart=/usr/local/bin/bookstore-db-backup

# /etc/systemd/system/bookstore-db-backup.timer
[Unit]
Description=Run the BookstoreDb backup every night

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=10min

[Install]
WantedBy=timers.target

Type=oneshot means "a command that does its job and exits". The service has no [Install] section because we do not start it, the timer does; what gets enabled is the timer. OnCalendar describes 02:30 every day, and RandomizedDelaySec adds a random delay between 0 and 10 minutes on top; that keeps many servers that back up at the same hour from hitting the database or storage at the same moment. Persistent=true writes the last trigger time to disk; if the server is off at 02:30, the timer runs the missed job right after boot. cron has no equivalent.

$ sudo systemctl enable --now bookstore-db-backup.timer
Created symlink /etc/systemd/system/timers.target.wants/bookstore-db-backup.timer → /etc/systemd/system/bookstore-db-backup.timer.
$ systemctl list-timers bookstore-db-backup.timer --no-pager
NEXT                        LEFT LAST PASSED UNIT                      ACTIVATES
Sun 2026-11-15 02:37:36 UTC  11h -         - bookstore-db-backup.timer bookstore-db-backup.service

1 timers listed.
Pass --all to see loaded but inactive timers, too.
$ systemd-analyze calendar "*-*-* 02:30:00" --iterations 2
Normalized form: *-*-* 02:30:00
    Next elapse: Sun 2026-11-15 02:30:00 UTC
       From now: 11h left
   Iteration #2: Mon 2026-11-16 02:30:00 UTC
       From now: 1 day 11h left
$ systemd-analyze calendar "Mon..Fri 08:00" "*:0/15"
  Original form: Mon..Fri 08:00
Normalized form: Mon..Fri *-*-* 08:00:00
    Next elapse: Mon 2026-11-16 08:00:00 UTC
       From now: 1 day 16h left

  Original form: *:0/15
Normalized form: *-*-* *:00/15:00
    Next elapse: Sat 2026-11-14 15:15:00 UTC
       From now: 1min 5s left

The next run is at 02:37:36: 02:30 plus the random delay. systemd-analyze calendar is the easiest way to try an expression before writing it; it normalizes expressions like weekdays at eight in the morning or every fifteen minutes and gives the next time. Times are in the server's time zone; this server uses UTC. To test without waiting for the night, start the service by hand:

$ sudo systemctl start bookstore-db-backup.service
$ sudo journalctl -u bookstore-db-backup.service --no-pager -o short | cut -c1-160
Nov 14 15:13:54 bookstore-dev systemd[1]: Starting bookstore-db-backup.service - Nightly BookstoreDb backup...
Nov 14 15:13:54 bookstore-dev bookstore-db-backup[2643]: backup started for BookstoreDb at 2026-11-14T15:13:54Z
Nov 14 15:13:54 bookstore-dev systemd[1]: bookstore-db-backup.service: Deactivated successfully.
Nov 14 15:13:54 bookstore-dev systemd[1]: Finished bookstore-db-backup.service - Nightly BookstoreDb backup.

The script's output landed in the journal without any redirection to a log file, and the run time and result are there too. The differences between timers and cron:

cron: crontab, Expressions and the Environment Trap

cron is still everywhere and handy for one-line jobs. Every user has their own crontab, edited with crontab -e; crontab -l lists it, crontab -r deletes it (without asking, careful). A line consists of five time fields and the command: minute, hour, day of month, month, day of week. 30 2 * * * means every day at 02:30, */15 * * * * every fifteen minutes, 0 8 * * 1-5 weekdays at 08:00.

cron's classic trap is that the command runs not in your shell's environment but in a much leaner one. I tried it with a crontab; I added the lines with crontab -e:

$ crontab -l
* * * * * env > /tmp/cron-env.txt
* * * * * bookstore-report >> /tmp/cron-report.log 2>&1
* * * * * /home/ada/bookstore-tools/count-books.sh >> /tmp/cron-count.log 2>&1
* * * * * cd /home/ada/bookstore-tools && ./count-books.sh >> /tmp/cron-count.log 2>&1
$ cat /tmp/cron-env.txt
HOME=/home/ada
LOGNAME=ada
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
LANG=C.UTF-8
SHELL=/bin/sh
PWD=/home/ada
$ cat /tmp/cron-report.log
/bin/sh: 1: bookstore-report: not found
$ cat /tmp/cron-count.log
/home/ada/bookstore-tools/count-books.sh: line 2: data/books.csv: No such file or directory
3

There are only six variables, and the command is run by /bin/sh, not bash. Older sources say cron's PATH is only /usr/bin:/bin; on Ubuntu 24.04 it is different: the pam_env line in /etc/pam.d/cron reads /etc/environment and PATH comes from there, /usr/local/bin included. That is why the backup script we put in /usr/local/bin ran from cron without trouble. But ~/.profile and ~/.bashrc are not read at all: the PATH addition that finds ~/.local/bin/bookstore-report in the shell, the exports in .bashrc and ~/.dotnet/tools for dotnet tools are missing in cron, and the command failed with "not found". The working folder is also the home folder: the script that reads data/books.csv relative to its own folder could not find the file when called with an absolute path, while the line that runs cd first counted 3 books.

The rule: write everything with absolute paths in cron; inside the script, either use absolute paths too or cd to its own folder, define the variables you need explicitly at the top of the crontab or inside the script, and always redirect the output somewhere. In files placed in /etc/cron.d/ for system-wide jobs, the sixth field is the user that runs the command. Because systemd timers solve most of these rules by themselves, we prefer them for new jobs.

Production Notes

Keep unit files, drop-ins and timers in the repository and copy them to the server as part of the deploy. For Bookstore that is the deploy/linux/systemd/ folder. A change made by hand on the server with systemctl edit becomes a change that disappears when the server is rebuilt and that nobody remembers. If you did make a change by hand, compare the systemctl cat output with the file in the repository.

Restart=always can hide a crash loop. If the application crashes every time it starts, systemd restarts it every 10 seconds and someone looking from the outside may think "the service is up". By default systemd gives up and marks the service failed if there are more than 5 start attempts within 10 seconds (StartLimitIntervalSec, StartLimitBurst); but because our unit has RestartSec=10, attempts are 10 seconds apart, that limit is never reached, and the loop goes on forever. Track how many times it restarted with systemctl show -p NRestarts and wire that into monitoring. Diagnosing a crash loop is the first case in the playbook of the twelfth post.

Common Pitfalls

Changing the unit file and forgetting daemon-reload. I lowered RestartSec from 10 to 5 and only restarted:

$ sudo sed -i "s/^RestartSec=10/RestartSec=5/" /etc/systemd/system/bookstore-api.service
$ sudo systemctl restart bookstore-api
Warning: The unit file, source configuration file or drop-ins of bookstore-api.service changed on disk. Run 'systemctl daemon-reload' to reload units.
$ systemctl show -p RestartUSec bookstore-api
RestartUSec=10s
$ sudo systemctl daemon-reload && systemctl show -p RestartUSec bookstore-api
RestartUSec=5s

systemd warned but still did the restart, with the old setting. If the warning gets lost in the output of a deploy script, hours can pass with "I changed it but it did not work". Every step that changes a unit file should be followed by daemon-reload; systemctl edit does it by itself.

Mistaking start for enable. As we saw above, the service is gone after a reboot. After installing a new service, check with systemctl is-enabled, or better, use enable --now from the start.

Assuming your shell's environment in cron. Do not assume a command that works in the shell will work in cron; try the crontab line once in a lean environment such as env -i HOME=$HOME LOGNAME=$USER PATH=/usr/bin:/bin sh -c '...', or run it on a one-minute schedule and look at its output.

Editing a packaged unit in /usr/lib/systemd/system. The next package update silently wipes the change. Always use a drop-in.

A Small Experiment

★☆☆ Install bookstore-api.service on your own VM, start it only with start first and reboot the VM. Then enable it, try again and look at the symlink under multi-user.target.wants.

★★☆ Create a drop-in with systemctl edit bookstore-api and add Environment=Logging__LogLevel__Default=Warning. See both files together with systemctl cat, restart the service and check that the log lines in the journal decreased. Then undo it with systemctl revert.

★★★ Install the backup timer, set OnCalendar to two minutes from now and shut the VM down before that minute. After the minute has passed, start the VM and verify with journalctl -u bookstore-db-backup that the job ran at boot thanks to Persistent=true. Set up the same job with cron and compare.

Bookstore is now in systemd's hands: it comes up at boot, restarts when it crashes, and the nightly backup runs on time. All of their output goes to the journal. In the next post we start reading logs: journalctl's filters, the classic files in /var/log, and logrotate to keep logs from filling the disk.