Blog
Mert Özen's blog posts on software development, backend, frontend, and DevOps.
A Linux Lab for Developers and SSH: Setting Up a VM, Connecting with Keys and Getting to Know the Server
You connect to a Linux server with an ed25519 SSH key instead of a password: generate it with ssh-keygen, install it with ssh-copy-id and define an alias in ~/.ssh/config. With real output and Windows equivalents, we cover ssh-agent, why the host key warning matters, a first tour with uname/df/free/ss, moving Bookstore.Api with rsync and reaching an app bound to 127.0.0.1 through an SSH tunnel.
The Linux Filesystem Hierarchy: /etc, /var, /opt, /srv, /home, /proc and "Everything Is a File"
On Linux, programs live in /usr or /opt, configuration in /etc and changing data in /var; for Bookstore that means /opt/bookstore/releases, /etc/bookstore, /var/log/bookstore and /var/lib/bookstore. With real output we cover find, cp -a and rm -r, hard links vs symlinks, how ln -sf silently creates the link in the wrong place when switching the current symlink, inspecting a running .NET process through /proc/<pid>, and why appsettings.production.json is ignored on Linux.
Users, Groups and Permissions: rwx, chmod, chown, umask, sudo and Service Accounts
On Linux you run an application neither as root nor as your own user, but as a service user that belongs to it alone and cannot log in; for Bookstore that is a bookstore user, a separate deploy user that writes releases, and a secrets.env with mode 640. With real output we cover what rwx means on directories, chmod/chown/umask, setgid directories, the sticky bit, ACLs, a sudoers rule written with visudo that allows a single command, and the pitfalls of chmod 777 and sudo.
Processes and Signals: ps, top/htop, kill, nice, the Process Tree and Zombies
If you stop a .NET application on Linux with SIGTERM or SIGINT, it finishes in-flight requests and exits cleanly; SIGHUP and SIGKILL kill it at once (shown with a real experiment). We cover reading ps and top output (VIRT/RES, STAT, load average), the process tree, which jobs die when SSH drops with jobs/fg/bg, the difference between nohup, disown and setsid, SigIgn, nice/renice, open files with /proc and lsof, and why zombies do not die with kill.
Managing Services with systemd: systemctl, Unit Overrides, Timers and Cron
To have a service come back after a reboot, start is not enough; you need systemctl enable, and Restart=always only applies when a running service's process dies (shown with a real reboot). We cover managing Bookstore.Api with systemctl, where unit files live, adding a drop-in override to Nginx with systemctl edit, After/Wants/Requires, systemd-analyze, a nightly backup timer with Persistent=true, and cron's real environment traps on Ubuntu 24.04.
Reading Logs 1: journalctl, /var/log and logrotate
You read a service's log with journalctl -u; -p err narrows it by level, --since by time and -b by boot, and if the user is not in the adm group the journal looks empty. We show reading Bookstore.Api's logs with journalctl filters, how AddSystemd carries .NET log levels into the journal, persistent vs volatile journal with a real reboot, auth.log, kern.log and the Nginx logs in /var/log, and create vs copytruncate in logrotate.
Reading Logs 2: Getting Answers from Logs with Pipes, Redirection, grep, awk, sed, sort and uniq
The way to get answers out of logs is to chain small tools with pipes: awk extracts a field, grep filters, sort orders, uniq -c counts. After the basics of stdin/stdout/stderr and redirection, we find the top IPs, 5xx per minute, the slowest requests and a request ID's trail in the application log on Bookstore's real Nginx access log, and show the uniq, regex and locale pitfalls.
Network Commands: ip, ss, ping, curl, dig and nc for "Is the Port Open, Who Is Listening, Why Can't I Reach It?"
When a service cannot be reached you go layer by layer: the name with getent, the path and firewall with ping and nc -zv (refused means nothing listens, timeout means packets are dropped), who listens on which address with ss -tlnp, then curl and the error log. We show ip, ss, lsof, curl, traceroute, dig and resolvectl on the Bookstore server, solve a real case where a stale hosts entry, ufw and a wrong port stack up, and look at proxy and IPv6 pitfalls.
Package Management: apt, Repositories, Updates and Installing .NET and Node on a Server
On Ubuntu 24.04 you install .NET from Ubuntu's own feed and keep packages.microsoft.com out with an apt pin file; apt-cache policy shows where each package comes from and with which priority. We show the basics of apt, deb822 and Signed-By, the backports PPA, installing NodeSource without the script, apt-mark hold, unattended-upgrades, and needrestart restarting Bookstore.Api during a real upgrade.
Disk, Memory and CPU: df, du, free, vmstat, iostat, the OOM Killer and the "Disk Full" Case
Deleting an open log file with rm does not free space; the process holds it, lsof +L1 shows it and truncate gives the space back. With a real case of Bookstore's log disk filling up, we show df, du and inodes, the available column and the page cache in free, the OOM killer and systemd's MemoryMax limit, reading the load average with the core count, and telling CPU from I/O waits with vmstat, pidstat and iostat.
Hardening a Server: SSH Settings, ufw, Fail2ban, Automatic Updates and Least Privilege
To turn off password logins on Ubuntu, put the setting in a drop-in under /etc/ssh/sshd_config.d/ that is read before cloud-init's file and verify it with sshd -T, because sshd uses the first value it reads. On the Bookstore server we show the second session rule, enabling ufw in the right order, Docker bypassing ufw for published ports, Fail2ban and least privilege with systemd, all with real output.
A Server Troubleshooting Playbook: From "The Site Is Down" to the Root Cause
When the site is down, end the outage first and then find the cause: rule out resources in 60 seconds with uptime, dmesg, free, df, systemctl --failed, journalctl -p err, ss and top, then narrow the hypothesis with the Nginx error log and the journal. With three Bookstore cases produced in the lab we solve a crash loop and 502 after a deploy, a backup job stealing CPU and an expired certificate, all with real output.