A Local Kubernetes Cluster and kubectl: kind, k3d, minikube, Contexts and Namespaces

Version note: The examples in this post were run with kind 0.33 on Kubernetes 1.37, kubectl 1.37 and Docker 29.

Guess First

You have two kind clusters on your machine: k8swork1-1, running Bookstore's three API (Application Programming Interface) pods, and k8swork1-2, an empty one you just created to try something. Without running any context command you type kubectl get pods -n bookstore. What's the output: three pods, an error, or something else? Second question: you loaded the bookstore-api:latest image into the cluster with kind load and created a Deployment with it. Does the pod reach Running?

Local Cluster Options

You don't need a cloud cluster to learn Kubernetes or to try your manifests. There are four common options, and all of them run on your own machine, mostly on top of Docker:

We use kind in this series, for three reasons. First, you can describe a multi-node cluster in a single YAML (YAML Ain't Markup Language) file; topics like scheduling, node failure and pod spread are invisible on a single node. Second, by choosing the node image you can install the Kubernetes version you want. Third, the same config runs on a CI runner; kind is widely used to test manifests against a real cluster in GitHub Actions. The project's own description is honest: kind was primarily designed for testing Kubernetes itself, but can be used for local development and CI.

A Multi-Node Cluster with kind

Downloading kind and kubectl into the project folder is enough; both are single binaries. The file that describes the cluster:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30080
        hostPort: 18780
        protocol: TCP
  - role: worker
  - role: worker

One control plane, two workers. extraPortMappings may look unimportant for now, but we'll need it in post 6 when we bring external traffic into the cluster: it binds port 30080 of the control plane container to port 18780 on the machine. Since kind nodes are Docker containers, this is the simplest way to reach the cluster from the host.

One more line before creating the cluster:

# make kind and kubectl use this file and leave ~/.kube/config alone
export KUBECONFIG=$PWD/kubeconfig
kind create cluster --name k8swork1-1 --config kind-bookstore.yaml \
  --image kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5

If KUBECONFIG isn't set, kind writes the cluster's access details into ~/.kube/config and switches the current context to the new cluster. If that file also holds your work clusters, you don't want that; we'll see why shortly. Pinning the node image by digest is what kind's release notes recommend: the same tag can be rebuilt for different kind versions. The end of the output:

 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-k8swork1-1"
You can now use your cluster with:

kubectl cluster-info --context kind-k8swork1-1
$ kubectl get nodes -o wide
NAME                       STATUS   ROLES           AGE   VERSION   INTERNAL-IP   OS-IMAGE                       CONTAINER-RUNTIME
k8swork1-1-control-plane   Ready    control-plane   41s   v1.37.0   172.20.0.2    Debian GNU/Linux 13 (trixie)   containerd://2.3.4
k8swork1-1-worker          Ready    <none>          27s   v1.37.0   172.20.0.3    Debian GNU/Linux 13 (trixie)   containerd://2.3.4
k8swork1-1-worker2         Ready    <none>          27s   v1.37.0   172.20.0.4    Debian GNU/Linux 13 (trixie)   containerd://2.3.4

A three-node cluster in about 20 seconds. To shut it down, kind delete cluster --name k8swork1-1; since the nodes are containers, nothing is left behind.

Loading a Local Image into the Cluster

You built the Bookstore images on your machine with docker build, but kind nodes don't see your Docker image cache; each node runs its own containerd. Instead of pushing to a registry, you can copy the image straight onto the nodes:

$ kind load docker-image bookstore-api:1.0.0 --name k8swork1-1
Image: "bookstore-api:1.0.0" with ID "sha256:aa62128f22bfc997e14b9f8822f01803ac90592c6a13d87655b7b4df1c40ae72" not yet present on node "k8swork1-1-control-plane", loading...

The command loads the image onto every node. Now the second question. I loaded the same image with the latest tag and created a Deployment:

$ docker tag bookstore-api:1.0.0 bookstore-api:latest
$ kind load docker-image bookstore-api:latest --name k8swork1-1
$ kubectl create deployment api-latest --image=bookstore-api:latest
deployment.apps/api-latest created
$ kubectl get deploy api-latest -o jsonpath='{.spec.template.spec.containers[0].imagePullPolicy}'
Always
$ kubectl get pods -l app=api-latest
NAME                          READY   STATUS         RESTARTS   AGE
api-latest-76677f88bf-bftt9   0/1     ErrImagePull   0          8s
$ kubectl get events --sort-by=.lastTimestamp
Normal    Pulling   pod/api-latest-76677f88bf-bftt9   Pulling image "bookstore-api:latest"
Warning   Failed    pod/api-latest-76677f88bf-bftt9   Failed to pull image "bookstore-api:latest": failed to pull and unpack image "docker.io/library/bookstore-api:latest": failed to resolve reference "docker.io/library/bookstore-api:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Warning   Failed    pod/api-latest-76677f88bf-bftt9   Error: ImagePullBackOff

The image was on the node, yet the pod didn't start. When the tag is :latest or missing, Kubernetes defaults imagePullPolicy to Always; the kubelet goes to the registry every time, and since the short name expands to docker.io/library/bookstore-api, it asks Docker Hub, where no such repository exists. With a version tag (1.0.0) the default is IfNotPresent and the image on the node is used. That's why we always use versioned tags in this series; in post 4 we'll see the production-side reason as well.

kubeconfig, Contexts and Namespaces

kubectl learns which cluster to talk to from the kubeconfig file. The file has three lists: clusters (API server address and CA (Certificate Authority) certificate), users (credentials) and contexts, which pair the two with a default namespace. The current-context field decides where every command without --context goes.

On to the first question. While the Bookstore cluster was running, I created a second kind cluster:

$ kind create cluster --name k8swork1-2 --image kindest/node:v1.37.0@sha256:a1ed56cf...
...
Set kubectl context to "kind-k8swork1-2"
$ kubectl config get-contexts
CURRENT   NAME              CLUSTER           AUTHINFO          NAMESPACE
          kind-k8swork1-1   kind-k8swork1-1   kind-k8swork1-1
*         kind-k8swork1-2   kind-k8swork1-2   kind-k8swork1-2
$ kubectl get pods -n bookstore
No resources found in bookstore namespace.

No error, exit code 0, and none of the three pods. Creating the new cluster silently changed the current context, and kubectl asked the empty cluster. That cluster doesn't even have a bookstore namespace, but kubectl doesn't report that as an error; it says "no resources in this namespace". Now picture the same situation with a kubectl delete or a kubectl apply: the command runs against the current context, not the cluster you have in mind.

Switching the context back and binding your usual namespace to it:

$ kubectl config use-context kind-k8swork1-1
Switched to context "kind-k8swork1-1".
$ kubectl config set-context --current --namespace=bookstore
Context "kind-k8swork1-1" modified.
$ kubectl config get-contexts
CURRENT   NAME              CLUSTER           AUTHINFO          NAMESPACE
*         kind-k8swork1-1   kind-k8swork1-1   kind-k8swork1-1   bookstore
          kind-k8swork1-2   kind-k8swork1-2   kind-k8swork1-2
$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
api-7dd799865f-85m2z   1/1     Running   0          11m
api-7dd799865f-j8nz5   1/1     Running   0          3m41s
api-7dd799865f-vftck   1/1     Running   0          12m
$ kubectl --context kind-k8swork1-2 get nodes
NAME                       STATUS     ROLES           AGE   VERSION
k8swork1-2-control-plane   NotReady   control-plane   11s   v1.37.0

The last command shows a one-off route: --context sends a command to another cluster without changing the current context. Writing it explicitly like this in scripts and CI is far safer than trusting the current context.

Namespaces give you a naming scope and a permission boundary inside the cluster: a Deployment called api can exist side by side in the bookstore-dev and bookstore-staging namespaces; RBAC (Role-Based Access Control) rules and ResourceQuotas are defined per namespace. But a namespace isn't a firewall: by default, pods in different namespaces can reach each other over the network. We'll close that with NetworkPolicy in post 15.

kubectl Commands You'll Use Every Day

kubectl get lists, -o wide adds the node and IP (Internet Protocol) columns, -o yaml shows the whole object. kubectl describe renders the object for humans and lists related events at the bottom; it's the first place to look when a pod won't start. kubectl explain reads documentation from the cluster's own schema, so the answer matches your version exactly:

$ kubectl explain deployment.spec.strategy.rollingUpdate.maxSurge
GROUP:      apps
KIND:       Deployment
VERSION:    v1

FIELD: maxSurge <IntOrString>

DESCRIPTION:
    The maximum number of pods that can be scheduled above the desired number of
    pods. Value can be an absolute number (ex: 5) or a percentage of desired
    pods (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is
    calculated from percentage by rounding up. Defaults to 25%.

Commands that change things have two safety nets. kubectl diff compares the manifest in the file with the live object in the cluster. --dry-run=server sends the request to the API server, runs it through all validation and admission steps, but doesn't persist it. To see the difference, I prepared a Deployment with an invalid name:

$ kubectl apply --dry-run=client -f bad2.yaml
deployment.apps/Bookstore_API created (dry run)
$ kubectl apply --dry-run=server -f bad2.yaml
The Deployment "Bookstore_API" is invalid: metadata.name: Invalid value: "Bookstore_API": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

The client-side dry run said "created", because it only parsed the YAML into an object. The server side caught that the name breaks the lowercase-and-dash rule. A misspelled field gets caught the same way: when I wrote replica instead of replicas, the server dry run returned strict decoding error: unknown field "spec.replica". Don't rely on client dry runs to validate manifests in a pipeline; use a server dry run or a test against a real cluster.

Version Skew Between kubectl and the Cluster

$ kubectl version
Client Version: v1.37.2
Kustomize Version: v5.8.1
Server Version: v1.37.0

Under Kubernetes' version skew policy, kubectl is supported within one minor version below or above the API server: for a 1.37 cluster that's 1.36, 1.37 and 1.38. If you keep a single kubectl on your laptop and connect to clusters ranging from 1.34 to 1.37, you're running an unsupported combination against the clusters at either end; most commands still work, but you'll see surprising behaviour around new or removed fields. Downloading a kubectl that matches the cluster version into the project folder solves the problem at the root. The Kustomize line in the output matters too: kubectl embeds a Kustomize version and kubectl apply -k uses it; we'll come back to that in post 14.

Production Notes

Don't put production cluster credentials into the kubeconfig you use every day. Keep a separate file per environment and select it only when needed, for that terminal, via KUBECONFIG. A prompt setting that shows the current context and namespace (tools like kube-ps1, or your own shell function) is cheap and effective insurance.

Minimise human write access to production. A read-only user for daily work, a pipeline for changes. Even when manual intervention is needed, run kubectl diff and --dry-run=server first.

If you use kind in CI, build the cluster from scratch in every job and delete it at the end; pin the node image by digest. Knowing which Kubernetes version your tests run against makes compatibility testing before an upgrade possible.

Common Pitfalls

Running commands in the wrong context. As you saw above, a tool (kind, a cloud CLI (Command Line Interface), an IDE (Integrated Development Environment) plugin) can change the current context behind your back. Write --context explicitly for delete and apply commands.

Creating drift with imperative commands. kubectl scale and kubectl set image are very handy while experimenting, but they don't change the manifest in your file. After running both on the Bookstore Deployment, kubectl diff showed the difference clearly:

$ kubectl scale deploy/api --replicas=5
deployment.apps/api scaled
$ kubectl set image deploy/api api=bookstore-api:1.1.0
deployment.apps/api image updated
$ kubectl diff -f api.yaml
...
 spec:
   progressDeadlineSeconds: 600
-  replicas: 5
+  replicas: 3
   revisionHistoryLimit: 10
...
     spec:
       containers:
-      - image: bookstore-api:1.1.0
+      - image: bookstore-api:1.0.0
         imagePullPolicy: IfNotPresent

Minus lines are the live state in the cluster, plus lines are what you'd get by applying the file. The next kubectl apply -f api.yaml pulls the replica count back to 3 and the image back to 1.0.0. A "temporary" kubectl scale done at night in production silently disappears with the next morning's deploy.

The :latest tag and imagePullPolicy. On a local cluster it's an ErrImagePull; in production a sneakier problem: the same tag can map to different images on different nodes.

Forgetting the namespace. A command without -n goes to the context's namespace, or to default if there isn't one. That's the answer to half of all "where are my pods?" questions.

A Small Experiment

★☆☆ Build your own kind cluster with the config above (set KUBECONFIG first), see the three nodes with kubectl get nodes -o wide, and find the same nodes as containers in the output of docker ps.

★★☆ Create two namespaces, bookstore-dev and bookstore-staging, and apply the same Deployment to each. Then define a separate context per namespace (kubectl config set-context dev --cluster=... --user=... --namespace=bookstore-dev) and compare the pod lists by switching contexts.

★★★ Open two terminals: point KUBECONFIG at one file in the first and at another file in the second, and create a separate kind cluster in each. Verify that changing the current context in one terminal doesn't affect the other. Then merge the two files with KUBECONFIG=a:b and interpret the output of kubectl config get-contexts.

We now have a running cluster and a kubectl that manages it safely. In the next post we go down to Kubernetes' smallest unit of execution: we write a pod spec for the Bookstore API and look at the non-root and port 8080 defaults of .NET container images, the pod lifecycle, init containers and native sidecars.