A Linux Lab for Developers and SSH: Setting Up a VM, Connecting with Keys and Getting to Know the Server

Version note: The commands in this post were run on a test VM with Ubuntu Server 24.04.5 LTS (kernel 6.8, OpenSSH 9.6p1, bash 5.2), from an Ubuntu 24.04 client with OpenSSH 9.6, and with the ASP.NET Core 10.0.15 runtime. The Windows PowerShell commands come from Microsoft's OpenSSH documentation; they were not run on Windows.

Guess First

You copied the API of Bookstore, the online bookstore used as the shared example across this site's series, to a freshly created Ubuntu server and started it. Asking from inside the server works:

ada@bookstore-dev:~$ curl -s http://127.0.0.1:5000/health
{"status":"Healthy"}

You send the same request from your own computer to the server's IP address:

$ curl -sS -m 5 http://10.20.30.10:5000/health
curl: (7) Failed to connect to 10.20.30.10 port 5000 after 0 ms: Couldn't connect to server

What is wrong: the firewall, the application, or neither? And is there a way to reach this API from the browser on your own machine without opening port 5000 to the outside? Both answers come near the end of the post, in an SSH tunnel.

This series is for developers who are not system administrators but still put their application on a Linux server. In this first post we set up a lab, connect to the server with a key instead of a password, take a first tour of the server and move files. Setting up the terminal and shell on your own machine was covered in the Terminal and Command Line series; here we are on the server side.

Lab Options: a VM, WSL 2 or the Cloud

Throughout the series we use Ubuntu Server 24.04 LTS; the differences that come with Ubuntu 26.04 are collected in the last post. WSL on Windows is also an option for the lab. You need a machine where you can try commands without fear of breaking something. Three realistic options:

My recommendation is a local VM. On Windows, one Multipass command is enough: multipass launch 24.04 --name bookstore-dev --cpus 2 --memory 2G --disk 20G. Multipass defaults to 1 CPU, 1 GB of memory and a 5 GB disk; we make it a bit bigger so the .NET runtime and Nginx run comfortably. Multipass sets up the VM with cloud-init and puts its own key inside; multipass shell bookstore-dev gets you in right away. In real life, though, you connect to servers with your own SSH key, so do the steps below anyway.

WSL 2 runs a real Linux kernel and is enough for most commands in this series. On the current Ubuntu installed with wsl --install, systemd is on by default, so systemctl works. But WSL does not behave like a server: it shares the network with Windows, nobody connects to it over SSH, and systemd services do not keep the WSL instance alive. Installing WSL is covered in the first post of the Terminal and Command Line series; in this series we mostly use WSL as the "client" side, because tools like ssh-copy-id come with it. A Docker container may also look like a lab, but a container does not run its own kernel and usually has no systemd inside; we covered the difference in the first post of the Docker series.

Connecting with a Key: ssh-keygen and ssh-copy-id

Connecting with a password works, but it has two problems: passwords can be guessed and you have to type one on every connection. An SSH key is a pair. The private key stays on your machine and is never shared; the public key is added to ~/.ssh/authorized_keys on the server. During the connection the server asks for a signature made with your private key and verifies it with the public key; the private key never crosses the network.

$ ssh-keygen -t ed25519 -C "ada@my-pc"
Generating public/private ed25519 key pair.
Your identification has been saved in /home/ada/.ssh/id_ed25519
Your public key has been saved in /home/ada/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:tYd+1iuceLFVUiYELrByeWsrkhqberfvyWYFK6epk0A ada@my-pc
$ ls -l ~/.ssh
-rw------- 1 ada ada 444 Nov 14 14:01 id_ed25519
-rw-r--r-- 1 ada ada  91 Nov 14 14:01 id_ed25519.pub

ed25519 is a short, fast and secure key type; since OpenSSH 9.5, ssh-keygen produces it even when run without parameters, and Microsoft's Windows documentation states the same default for ssh-keygen.exe on Windows. -C is just a comment; it helps you tell where a key came from when you look at authorized_keys on the server. Do not skip the passphrase: if the private key file is stolen, the passphrase makes it useless on its own. Look at the file permissions too: the private key was created as -rw------- (600), readable only by its owner.

On Windows the same command works in PowerShell; from Windows 10 (1809) on, OpenSSH Client ships as a Windows feature, check it with ssh -V. The keys are written under C:\Users\user\.ssh\.

The shortest way to put the public key on the server is ssh-copy-id. For that you need to be able to log in with a password once:

$ ssh-copy-id ada@10.20.30.10
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ada/.ssh/id_ed25519.pub"
The authenticity of host '10.20.30.10 (10.20.30.10)' can't be established.
ED25519 key fingerprint is SHA256:Yn16rbVCEQjPr1Vmyyfc/SVWZTYwiT5ye2Bb7mFEUXs.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
ada@10.20.30.10's password:

Number of key(s) added: 1

The "authenticity of host" question here is a separate topic and we will come back to it below. Windows PowerShell has no ssh-copy-id. If you use WSL 2, the Ubuntu there has it; if you stay in PowerShell, you can pipe the public key and append it to the file on the server:

type $env:USERPROFILE\.ssh\id_ed25519.pub | ssh ada@10.20.30.10 "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

The quoted part runs on the server. I wondered whether Windows line endings (CRLF) would be a problem: I appended a key line ending in \r\n with the same command and logged in with that key, and OpenSSH 9.6 accepted it without complaint. With cloud providers and Multipass you usually skip this step: you hand over the public key when you create the VM, cloud-init puts it in place, and password login is already off.

~/.ssh/config and ssh-agent

Typing ssh ada@10.20.30.10 every time is long and error-prone. Add a host alias to ~/.ssh/config (on Windows, C:\Users\user\.ssh\config):

Host bookstore-dev
    HostName 10.20.30.10
    User ada
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes

Now ssh bookstore-dev is enough; the same alias works in every tool that uses SSH, such as scp, rsync and the Remote-SSH extension of Visual Studio Code. IdentitiesOnly yes stops SSH from trying every key it has one after another; if you have many keys, the server can drop you with "Too many authentication failures". To see which settings SSH ends up using for a host, ssh -G helps:

$ ssh -G bookstore-dev | grep -E "^(hostname|user|port|identityfile|identitiesonly) "
user ada
hostname 10.20.30.10
port 22
identitiesonly yes
identityfile ~/.ssh/id_ed25519

If you gave the key a passphrase, it is asked on every connection. ssh-agent keeps the private key decrypted in memory and signs for the SSH client:

$ ssh bookstore-dev hostname
Enter passphrase for key '/home/ada/.ssh/id_ed25519':
bookstore-dev
$ eval "$(ssh-agent -s)"
Agent pid 503
$ ssh-add
Identity added: /home/ada/.ssh/id_ed25519 (ada@my-pc)
$ ssh-add -l
256 SHA256:tYd+1iuceLFVUiYELrByeWsrkhqberfvyWYFK6epk0A ada@my-pc (ED25519)
$ ssh bookstore-dev hostname
bookstore-dev

On Windows, ssh-agent is a Windows service and it is disabled by default. The order in Microsoft's documentation is: in an elevated PowerShell run Get-Service ssh-agent | Set-Service -StartupType Automatic, then Start-Service ssh-agent, then as your normal user ssh-add $env:USERPROFILE\.ssh\id_ed25519. The service keeps the key with your Windows session, so you do not need ssh-add again after a reboot.

Why the Host Key Warning Matters

The "authenticity of host can't be established" question on the first connection is about the server proving who it is. Your key proves to the server who you are; the server's host key proves to you that the other side really is that server. When you answer yes, the fingerprint is written to ~/.ssh/known_hosts and compared on every later connection. The right way is to verify the fingerprint through another channel: the VM console, the cloud provider's serial console, or, with Multipass, multipass exec to look at the host key on the server.

ada@bookstore-dev:~$ ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
256 SHA256:OPGgW2jQqmd36iAMuiWEl3UFEwdxO47/mRo1GI/gFSI root@bookstore-dev (ED25519)

This output is from after I regenerated the server's host keys; it is exactly what happens when you delete a VM and recreate it with the same IP. On the next connection SSH said this and refused to connect:

$ ssh bookstore-dev hostname
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:OPGgW2jQqmd36iAMuiWEl3UFEwdxO47/mRo1GI/gFSI.
...
Offending ECDSA key in /home/ada/.ssh/known_hosts:3
  remove with:
  ssh-keygen -f '/home/ada/.ssh/known_hosts' -R '10.20.30.10'
Host key verification failed.

This message has two explanations: either you rebuilt the server, or someone got in between (a man-in-the-middle attacker who listens to and changes the traffic). Deleting known_hosts without knowing which one it is means switching off the alarm when it rings. If you rebuilt the server and the new fingerprint matches what you saw on the console, remove the old entry and connect again:

$ ssh-keygen -R 10.20.30.10
# Host 10.20.30.10 found: line 1
# Host 10.20.30.10 found: line 2
# Host 10.20.30.10 found: line 3
/home/ada/.ssh/known_hosts updated.
Original contents retained as /home/ada/.ssh/known_hosts.old
$ ssh -o StrictHostKeyChecking=accept-new bookstore-dev hostname
Warning: Permanently added '10.20.30.10' (ED25519) to the list of known hosts.
bookstore-dev

accept-new stores new hosts without asking but still rejects a changed key, which makes it much safer than the StrictHostKeyChecking=no often used in CI. We saw GitHub's host key and known_hosts in the Git & GitHub series post "Connecting to Your GitHub Account"; it is the same mechanism.

Getting to Know the Server on the First Tour

When you log in to a new server, the first five minutes go to questions like "what is this machine, what runs on it, how much space does it have". In order:

$ uname -a
Linux bookstore-dev 6.8.0-147-generic #147-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep 24 11:33:16 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
$ grep -E "^(PRETTY_NAME|VERSION_CODENAME)=" /etc/os-release
PRETTY_NAME="Ubuntu 24.04.5 LTS"
VERSION_CODENAME=noble
$ hostnamectl
 Static hostname: bookstore-dev
       Icon name: computer-vm
         Chassis: vm
  Virtualization: qemu
Operating System: Ubuntu 24.04.5 LTS
          Kernel: Linux 6.8.0-147-generic
    Architecture: x86-64
 Hardware Vendor: QEMU
$ uptime
 14:05:54 up 1 min,  1 user,  load average: 2.53, 1.16, 0.44
$ whoami; id
ada
uid=1000(ada) gid=1000(ada) groups=1000(ada),27(sudo)

uname gives the kernel version and the architecture; this is where you learn whether to publish .NET for linux-x64 or linux-arm64. /etc/os-release names the distribution and its version; it is where you see the name "noble" you will meet when installing packages. hostnamectl puts these on one screen and also shows whether the machine is a VM or a container (I shortened the Machine ID and firmware lines in the output). We will read the load average from uptime in detail in the fourth post; here we only see that the machine has just booted. 27(sudo) in the id output means this user can get root privileges with sudo; users, groups and sudo are the topic of the third post.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           197M 1008K  196M   1% /run
/dev/vda1        19G  1.7G   17G   9% /
tmpfs           984M     0  984M   0% /dev/shm
/dev/vda16      881M   64M  756M   8% /boot
/dev/vda15      105M  6.2M   99M   6% /boot/efi
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       297Mi       1.5Gi       1.0Mi       313Mi       1.6Gi
Swap:             0B          0B          0B

The line to look at in df -h is /: 1.7 GB of 19 GB is used. Most "disk full" incidents in production start here, and in the tenth post we solve one from start to finish. In free -h, look at the "available" column, not "free"; Linux uses idle memory for the file cache and gives it back when needed. This VM has no swap either, which is common on cloud images.

$ ss -tlnp
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      4096         0.0.0.0:22        0.0.0.0:*
LISTEN 0      4096   127.0.0.53%lo:53        0.0.0.0:*
LISTEN 0      4096      127.0.0.54:53        0.0.0.0:*
LISTEN 0      4096            [::]:22           [::]:*
$ sudo ss -tlnp
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      4096         0.0.0.0:22        0.0.0.0:*    users:(("sshd",pid=1053,fd=3),("systemd",pid=1,fd=58))
LISTEN 0      4096   127.0.0.53%lo:53        0.0.0.0:*    users:(("systemd-resolve",pid=520,fd=15))
LISTEN 0      4096      127.0.0.54:53        0.0.0.0:*    users:(("systemd-resolve",pid=520,fd=17))
LISTEN 0      4096            [::]:22           [::]:*    users:(("sshd",pid=1053,fd=4),("systemd",pid=1,fd=59))

ss -tlnp answers "who listens on which TCP port on this machine": -t for TCP, -l for listeners only, -n for numeric ports instead of names, -p to show the process. Two lines stand out. First, without sudo the Process column stays empty; you need privileges to see other users' processes. Second, port 22 is held both by sshd and by systemd with PID 1. Since Ubuntu 22.10, SSH comes with socket activation: systemd opens the port and starts sshd when a connection arrives. The listener on 127.0.0.53 is Ubuntu's local DNS resolver. Ports and bind addresses in detail come in the eighth post.

When you cannot remember a command, there are three sources. ls --help gives a short summary. man ss is the full documentation; / searches inside it and q quits. man -k ^ssh lists every man page whose name starts with ssh and saves you when you cannot remember a tool's name. For short, example-driven pages there is the tldr project, but be careful: I tried both clients in the Ubuntu 24.04 repository (tldr 0.9.2 and tealdeer 1.6.1), and neither could download the page archive. The reason is that the tldr.sh/assets/tldr.zip address the old clients download now returns an HTML redirect page instead of a zip. You need to read the pages on the tldr.sh site or use a newer client.

Moving Files and SSH Tunnels

The trimmed-down Bookstore.Api for this series serves only the /health and /api/books endpoints. We publish it on our own machine with dotnet publish src/Bookstore.Api -c Release -r linux-x64 --self-contained false -o out/publish and move the output to the server. The ASP.NET Core runtime must be installed on the server (sudo apt install aspnetcore-runtime-10.0; package management is in the ninth post).

$ rsync -avz out/publish/ bookstore-dev:~/bookstore-api/
sending incremental file list
created directory /home/ada/bookstore-api
./
Bookstore.Api
Bookstore.Api.deps.json
Bookstore.Api.dll
Bookstore.Api.pdb
Bookstore.Api.runtimeconfig.json
Bookstore.Api.staticwebassets.endpoints.json

sent 51,191 bytes  received 179 bytes  9,340.00 bytes/sec
total size is 109,009  speedup is 2.12
$ rsync -avz out/publish/ bookstore-dev:~/bookstore-api/
sending incremental file list

sent 181 bytes  received 12 bytes  42.89 bytes/sec
total size is 109,009  speedup is 564.81

On the second run no file was sent: rsync compares the files on both sides and transfers only the difference. -a keeps permissions and timestamps, -z compresses. The trailing / on the source path matters: out/publish/ copies the folder's contents, out/publish copies the folder itself. For a single file, scp out/publish/Bookstore.Api.dll bookstore-dev:/tmp/ is enough; since OpenSSH 9.0, scp uses SFTP under the hood, and if an old server is incompatible, -O falls back to the legacy protocol. On Windows scp is built in but rsync is not; WSL 2 is the easiest way to get rsync.

Now the answer to the guess. On the server I started the application bound only to the loopback address:

ada@bookstore-dev:~$ cd ~/bookstore-api && nohup dotnet Bookstore.Api.dll --urls http://127.0.0.1:5000 > ~/api.log 2>&1 &

A process bound to 127.0.0.1 accepts only connections coming from inside the same machine. A request from outside fails with "Couldn't connect" before it even reaches a firewall, because nobody listens on that address. This is not a bug but the intended behavior: an application that will sit behind a reverse proxy such as Nginx should not be exposed. So how do you, as a developer, reach it? With SSH local port forwarding:

$ ssh -f -N -L 5000:127.0.0.1:5000 bookstore-dev
$ curl -s http://localhost:5000/api/books
[{"isbn":"9780441172719","title":"Dune","price":14.99},{"isbn":"9780141439587","title":"Emma","price":9.50},{"isbn":"9780547928227","title":"The Hobbit","price":12.00}]

-L 5000:127.0.0.1:5000 says: listen on 5000 on my machine, carry every connection through SSH to the server and connect it to 127.0.0.1:5000 as seen from the server. -N means do not run a remote command, -f means go to the background. When I closed the tunnel (stopped the ssh process), the same request returned "Couldn't connect" again. You reach a SQL Server that only accepts connections from localhost on the server the same way: ssh -N -L 14333:127.0.0.1:1433 bookstore-dev, then Server=127.0.0.1,14333 in SSMS or in the connection string. The database port is never exposed to the internet, and access is as safe as your SSH key. You can also tunnel to another machine the server can reach on 1433: in -L 14333:db-host:1433 the target address is resolved from the server's point of view.

Production Notes

Use personal keys on production servers and do not share them. Everyone on the team should be a separate line in authorized_keys; when someone leaves, you delete only their line. Passing a shared "deploy key" around in Slack means you cannot tell who did what and cannot cut off access for someone who left. For deploys from CI, use a separate key that is allowed to do only that job; the GitHub Actions series post on deploying over SSH shows that setup.

On this VM, cloud-init allowed password login through its ssh_pwauth setting; sshd -T shows the effective value:

ada@bookstore-dev:~$ sudo sshd -T | grep -E "^(passwordauthentication|permitrootlogin|pubkeyauthentication) "
permitrootlogin without-password
pubkeyauthentication yes
passwordauthentication yes

Turning off password login after you have seen key login work is one of the first things to do on an internet-facing server. The order of the files under /etc/ssh/sshd_config.d/ and the "second session" rule that keeps you from locking yourself out come in the eleventh post. And do not forget tunnels: ssh -f -L processes left running in the background keep holding the port; when you try to open a second tunnel on the same port you get bind [127.0.0.1]:5000: Address already in use.

Common Pitfalls

Leaving the private key's permissions loose. When you copy the key from somewhere or extract it from a zip, the permissions can end up as 644, and OpenSSH refuses to use the key:

$ chmod 644 ~/.ssh/id_ed25519 && ssh bookstore-dev hostname
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home/ada/.ssh/id_ed25519' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/home/ada/.ssh/id_ed25519": bad permissions
ada@10.20.30.10: Permission denied (publickey,password).

The fix is chmod 600 ~/.ssh/id_ed25519. Note the last line: the error is "bad permissions", but the connection ends with "Permission denied"; read the lines above before you start looking for the problem on the server.

Logging in as root with a password. Ubuntu's default already blocks password login for root (permitrootlogin without-password), and enabling it hands the guessing attempts that hit every internet-facing server a fully privileged target. Log in with your own user and use sudo when needed.

Sharing the private key, or putting the private key on the server instead of the public one. The file that goes into authorized_keys is always the one ending in .pub. Once a private key has gone somewhere else, treat it as leaked: generate a new pair and remove the old public key from every server.

Clicking through the host key warning out of habit. StrictHostKeyChecking=no, or deleting known_hosts at every warning, turns off the only man-in-the-middle protection SSH gives you.

A Small Experiment

★☆☆ Create an Ubuntu Server 24.04 VM named bookstore-dev with Multipass or Hyper-V. Generate your ed25519 key, put it on the server, add an alias to ~/.ssh/config and connect without a password using ssh bookstore-dev. Run the first-tour commands and note your VM's kernel version, disk size and listening ports.

★★☆ Publish Bookstore.Api, move it to the server with rsync and run it on 127.0.0.1:5000. First see that you cannot reach it from outside, then that you can open http://localhost:5000/api/books in your browser through an ssh -L tunnel. Make the same tunnel permanent with a LocalForward 5000 127.0.0.1:5000 line in ~/.ssh/config.

★★★ Delete the VM and recreate it with the same IP (or regenerate the host keys on the server) and see the REMOTE HOST IDENTIFICATION HAS CHANGED warning yourself. Do not touch known_hosts before verifying the new fingerprint from the VM console; once verified, clean the old entry with ssh-keygen -R.

You now log in to the server with a key, know what it is and can move files. In the next post we walk around inside the server: what /etc, /var, /opt and /proc are for, where Bookstore's files should go, and how release folders work with a current symlink.