Extending Kubernetes: CRDs, Controllers and the Operator Pattern, ownerReferences and Finalizers

Version note: The examples in this post were run with kind 0.33 on Kubernetes 1.37, kubectl 1.37, .NET 10 and KubernetesClient 21.0.8; the state of the operator libraries and the cert-manager version are as of November 2026.

Guess First

A namespace holds a single custom resource managed by a controller you wrote, and the object carries a finalizer that controller added. The controller's pod is not running for some reason. You run kubectl delete namespace. How many seconds until the namespace is gone? The "empty the finalizers with a patch" command you find online deletes the namespace right away; what does that command make you lose? We answer both questions in the second half of this post with our own controller.

The Reconciliation Loop in General

In the first post of the series we saw desired state and the reconciliation loop: you write the state you want, and a controller keeps comparing the actual state with it and closing the gap. The Deployment controller manages ReplicaSets this way, the ReplicaSet controller manages pods, the EndpointSlice controller manages endpoints. This post is about extending the same mechanism to types you add to Kubernetes yourself. We used it throughout the series without noticing: the Gateway and HTTPRoute from the sixth post, KEDA's ScaledObject from the eleventh, cert-manager's Certificate in the fifth post of the TLS series and Argo CD's Application in the GitOps series are not Kubernetes' own types; each is a CRD plus a controller that watches it.

The operator pattern is the name for that combination: a new type plus a controller that manages it and encodes the knowledge of running an application. cert-manager carries the knowledge of "how a certificate is obtained and renewed", CloudNativePG of "how a PostgreSQL cluster is set up, backed up and failed over".

A New Resource Type with a CRD

Bookstore, the online bookstore used as the shared example across this site's series, has campaigns: a title, a discount, a start and end time, and the ISBNs they cover. We define this in Kubernetes as a type called Campaign:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: campaigns.bookstore.example.com
spec:
  group: bookstore.example.com
  scope: Namespaced
  names:
    kind: Campaign
    plural: campaigns
    singular: campaign
    shortNames: ["camp"]
  versions:
    - name: v1alpha1
      served: true
      storage: true
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Discount
          type: integer
          jsonPath: .spec.discountPercent
        - name: Phase
          type: string
          jsonPath: .status.phase
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["title", "discountPercent", "startsAt", "endsAt"]
              x-kubernetes-validations:
                - rule: "self.endsAt > self.startsAt"
                  message: "endsAt must be after startsAt"
              properties:
                title:
                  type: string
                  maxLength: 80
                discountPercent:
                  type: integer
                  minimum: 1
                  maximum: 70
                startsAt:
                  type: string
                  format: date-time
                endsAt:
                  type: string
                  format: date-time
                isbns:
                  type: array
                  maxItems: 50
                  items:
                    type: string
                    pattern: "^97[89][0-9]{10}$"
            status:
              type: object
              properties:
                phase:
                  type: string
                observedGeneration:
                  type: integer
                  format: int64
                configMapName:
                  type: string

$ kubectl apply -f campaign-crd.yaml
customresourcedefinition.apiextensions.k8s.io/campaigns.bookstore.example.com created
$ kubectl get crd
NAME                              SCOPE        VERSIONS            CREATED AT
campaigns.bookstore.example.com   Namespaced   v1alpha1(storage)   2026-11-15T07:39:29Z
$ kubectl api-resources --api-group=bookstore.example.com
NAME        SHORTNAMES   APIVERSION                       NAMESPACED   KIND
campaigns   camp         bookstore.example.com/v1alpha1   true         Campaign

The API server now serves a new endpoint under /apis/bookstore.example.com/v1alpha1; kubectl get, explain, RBAC and watch all work for this type. The group name should be a domain you own; for the example I used the reserved example.com. The schema is not just documentation but validation: OpenAPI rules work per field, while x-kubernetes-validations lets you write cross-field rules in CEL (GA since 1.29):

$ kubectl apply -f campaign-invalid.yaml
The Campaign "broken" is invalid:
* spec.discountPercent: Invalid value: 90: spec.discountPercent in body should be less than or equal to 70
* spec.isbns[0]: Invalid value: "12345": spec.isbns[0] in body should match '^97[89][0-9]{10}$'
* spec: Invalid value: endsAt must be after startsAt

The invalid campaign never reached etcd; the controller does not need to check these. subresources.status moves status to a separate endpoint: the user writes spec, the controller writes status, and RBAC can authorize the two separately. There is another effect: metadata.generation only increases when spec changes, and the controller writes which generation it has processed to status.observedGeneration. When the two are equal, the controller has seen the latest change.

ownerReferences and Garbage Collection

When an object was created by another object, it carries its owner in metadata.ownerReferences. The easiest way to see this is the tree plugin we installed with krew in the previous post:

$ kubectl tree -n bookstore deployment api
NAMESPACE  NAME                          READY  REASON  STATUS   AGE
bookstore  Deployment/api                -              -        1s
bookstore  └─ReplicaSet/api-7dc5dc5b47   -              -        1s
bookstore    ├─Pod/api-7dc5dc5b47-4mz8g  True           Current  1s
bookstore    ├─Pod/api-7dc5dc5b47-95q9t  True           Current  1s
bookstore    └─Pod/api-7dc5dc5b47-z8w8h  True           Current  1s
$ kubectl -n bookstore get rs -l app.kubernetes.io/name=api -o jsonpath='{.items[0].metadata.ownerReferences}{"\n"}'
[{"apiVersion":"apps/v1","blockOwnerDeletion":true,"controller":true,"kind":"Deployment","name":"api","uid":"<uid>"}]

When the owner is deleted, the garbage collector deletes the objects left without an owner. With the default background deletion the owner goes immediately and the dependents follow. --cascade=orphan leaves the dependents behind, --cascade=foreground deletes the dependents first:

$ kubectl -n bookstore delete deploy api --cascade=orphan
deployment.apps "api" deleted from bookstore namespace
$ kubectl -n bookstore get rs,pods -l app.kubernetes.io/name=api
NAME                             DESIRED   CURRENT   READY   AGE
replicaset.apps/api-7dc5dc5b47   3         3         3       2s

NAME                       READY   STATUS    RESTARTS   AGE
pod/api-7dc5dc5b47-4mz8g   1/1     Running   0          2s
pod/api-7dc5dc5b47-95q9t   1/1     Running   0          2s
pod/api-7dc5dc5b47-z8w8h   1/1     Running   0          2s
$ kubectl -n bookstore apply -f api.yaml
deployment.apps/api created
$ kubectl -n bookstore get rs -l app.kubernetes.io/name=api -o 'custom-columns=NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].name,READY:.status.readyReplicas'
NAME             OWNER   READY
api-7dc5dc5b47   api     3
$ kubectl -n bookstore delete deploy api --cascade=foreground --wait=false
deployment.apps "api" deleted from bookstore namespace
$ kubectl -n bookstore get deploy api -o jsonpath='{.metadata.finalizers} {.metadata.deletionTimestamp}{"\n"}'
["foregroundDeletion"] 2026-11-15T07:41:35Z

Two details matter. When the same Deployment was created again, its controller adopted the orphaned ReplicaSet; no new pods were started and the pod names stayed the same. That is how you recreate a Deployment without downtime. With foreground deletion the Deployment was not deleted; it got a deletionTimestamp and a foregroundDeletion finalizer, and it waits until its dependents are gone. We will see what a finalizer is with our own controller shortly.

A Small Controller in .NET

Bookstore.CampaignController is a .NET 10 console app using KubernetesClient 21.0.8, the official C# client. For each Campaign it produces a ConfigMap holding the campaign's details (the API can mount and read it) and writes the banner to a storefront cache outside the cluster. In the lab that cache is a JSON file; in real life it would be a CDN, a search index or a campaign record at a payment provider. The structure follows the path real controllers take: the watch only says "look at this name again" and puts the name on a queue, and reconcile reads the current state of the object and decides what to do.

// Work queue: the watch only says "look at this name again"; the reconciler decides what to do.
Channel<string> queue = Channel.CreateUnbounded<string>();

Task worker = Task.Run(async () =>
{
    await foreach (string name in queue.Reader.ReadAllAsync(cts.Token))
    {
        try
        {
            await reconciler.ReconcileAsync(ns, name, cts.Token);
        }
        catch (Exception e) when (e is not OperationCanceledException)
        {
            CampaignReconciler.Log($"{ns}/{name}: reconcile failed ({e.GetType().Name}), retrying in 5s");
            _ = Task.Delay(TimeSpan.FromSeconds(5), cts.Token).ContinueWith(_ => queue.Writer.TryWrite(name), TaskScheduler.Default);
        }
    }
});

CampaignReconciler.Log($"watching campaigns in namespace {ns}");
try
{
    while (!cts.IsCancellationRequested)
    {
        // A watch ends after a server-side timeout; re-listing on every loop also covers missed events.
        try
        {
            await foreach ((WatchEventType type, object item) in client.CustomObjects.WatchListNamespacedCustomObjectAsync(
                Campaign.Group, Campaign.Version, ns, Campaign.Plural, timeoutSeconds: 300, cancellationToken: cts.Token))
            {
                Campaign campaign = KubernetesJson.Deserialize<Campaign>(((JsonElement)item).GetRawText());
                CampaignReconciler.Log($"event {type} {campaign.Metadata.Name} (generation {campaign.Metadata.Generation})");
                queue.Writer.TryWrite(campaign.Metadata.Name);
            }
        }
        catch (HttpOperationException e)
        {
            // 404 here usually means the CRD is not installed (yet or any more): wait instead of crashing.
            CampaignReconciler.Log($"watch failed ({(int)e.Response.StatusCode}), retrying in 10s");
            await Task.Delay(TimeSpan.FromSeconds(10), cts.Token);
        }
    }
}
catch (OperationCanceledException)
{
}

Reconcile itself is "level-triggered": it does not look at which event arrived, it asks the same question every time: "what should exist right now, and what does?"

    // Level-triggered: always read the current object and move the world towards it.
    public async Task ReconcileAsync(string ns, string name, CancellationToken ct)
    {
        Campaign campaign;
        try
        {
            campaign = await client.CustomObjects.GetNamespacedCustomObjectAsync<Campaign>(
                Campaign.Group, Campaign.Version, ns, Campaign.Plural, name, ct);
        }
        catch (HttpOperationException e) when (e.Response.StatusCode == HttpStatusCode.NotFound)
        {
            return; // already gone; the ConfigMap is removed by garbage collection
        }

        string key = $"{ns}/{name}";
        List<string> finalizers = campaign.Metadata.Finalizers?.ToList() ?? [];

        if (campaign.Metadata.DeletionTimestamp is not null)
        {
            if (finalizers.Remove(Finalizer))
            {
                await storefront.RemoveAsync(key, ct);
                Log($"{key}: banner removed from storefront, releasing finalizer");
                await PatchFinalizersAsync(ns, name, finalizers, ct);
            }
            return;
        }

        if (!finalizers.Contains(Finalizer))
        {
            finalizers.Add(Finalizer);
            await PatchFinalizersAsync(ns, name, finalizers, ct);
            Log($"{key}: finalizer added");
            return; // the update triggers another reconcile
        }

        string configMapName = $"campaign-{name}";
        await EnsureConfigMapAsync(campaign, configMapName, ct);
        await storefront.PublishAsync(key, campaign.Spec.Title, ct);

        DateTimeOffset now = clock.GetUtcNow();
        string phase = now < campaign.Spec.StartsAt ? "Scheduled" : now >= campaign.Spec.EndsAt ? "Expired" : "Active";
        long generation = campaign.Metadata.Generation ?? 0;
        if (campaign.Status is not { } status || status.Phase != phase || status.ObservedGeneration != generation)
        {
            var patch = new { status = new CampaignStatus(phase, generation, configMapName) };
            await client.CustomObjects.PatchNamespacedCustomObjectStatusAsync(
                new V1Patch(patch, V1Patch.PatchType.MergePatch), Campaign.Group, Campaign.Version, ns, Campaign.Plural, name,
                cancellationToken: ct);
            Log($"{key}: generation {generation} reconciled, phase {phase}");
        }
    }

EnsureConfigMapAsync creates the ConfigMap with the Campaign's name and UID in its ownerReferences (the full code is in the snapshot). In the lab I built the controller and ran it from outside the cluster with the kubeconfig (dotnet bin/Debug/net10.0/Bookstore.CampaignController.dll); inside the cluster the same code uses the ServiceAccount token. I created a campaign and changed its discount:

$ kubectl -n bookstore get campaigns
NAME            DISCOUNT   PHASE    AGE
classics-week   20         Active   4s
$ kubectl -n bookstore get configmap campaign-classics-week -o jsonpath='{.metadata.ownerReferences}{"\n"}{.data}{"\n"}'
[{"apiVersion":"bookstore.example.com/v1alpha1","blockOwnerDeletion":true,"controller":true,"kind":"Campaign","name":"classics-week","uid":"<uid>"}]
{"discountPercent":"20","endsAt":"2026-11-30T23:59:59.0000000+00:00","isbns":"9780141439518,9780141439549","startsAt":"2026-11-10T00:00:00.0000000+00:00","title":"Classics Week"}
$ kubectl -n bookstore patch campaign classics-week --type=merge -p '{"spec":{"discountPercent":25}}'
campaign.bookstore.example.com/classics-week patched
$ kubectl -n bookstore get campaign classics-week -o jsonpath='{.metadata.generation} {.status}{"\n"}'
2 {"configMapName":"campaign-classics-week","observedGeneration":2,"phase":"Active"}

07:39:42 watching campaigns in namespace bookstore
07:39:46 event Added classics-week (generation 1)
07:39:46 event Modified classics-week (generation 1)
07:39:46 bookstore/classics-week: finalizer added
07:39:46 bookstore/classics-week: configmap campaign-classics-week created
07:39:46 bookstore/classics-week: generation 1 reconciled, phase Active
07:39:46 event Modified classics-week (generation 1)
07:39:50 event Modified classics-week (generation 2)
07:39:50 bookstore/classics-week: configmap campaign-classics-week updated
07:39:50 event Modified classics-week (generation 2)
07:39:50 bookstore/classics-week: generation 2 reconciled, phase Active

Note in the log that each change we write ourselves (the finalizer, the status) produces a new Modified event; reconcile passes over them without doing extra work, because the state already is the desired state. The status patch did not increase the generation; the spec patch did.

Finalizers: Cleaning Up Before Deletion

A finalizer is a name in the object's metadata.finalizers list; until the list is empty the API server does not delete the object, it only sets deletionTimestamp. When the controller sees that mark, it does the cleanup outside the cluster and removes its own finalizer from the list. The ConfigMap needs no finalizer; ownerReferences and the garbage collector handle it. A finalizer exists for things Kubernetes cannot see: here, the banner in the storefront.

$ kubectl -n bookstore delete campaign classics-week
campaign.bookstore.example.com "classics-week" deleted from bookstore namespace
$ kubectl -n bookstore get configmap campaign-classics-week
Error from server (NotFound): configmaps "campaign-classics-week" not found
$ cat storefront-banners.json; echo
{}

07:40:06 event Modified classics-week (generation 3)
07:40:06 bookstore/classics-week: banner removed from storefront, releasing finalizer
07:40:06 event Deleted classics-week (generation 3)

A Namespace Stuck in Terminating

The answer to Guess First. I created a campaign in the bookstore-promo namespace, stopped the controller after it had added the finalizer, and deleted the namespace:

$ kubectl delete namespace bookstore-promo --wait=false
namespace "bookstore-promo" deleted
$ kubectl get namespace bookstore-promo
NAME              STATUS        AGE
bookstore-promo   Terminating   25s
$ kubectl get namespace bookstore-promo -o jsonpath='{range .status.conditions[*]}{.type}={.status}: {.message}{"\n"}{end}'
NamespaceDeletionDiscoveryFailure=False: All resources successfully discovered
NamespaceDeletionGroupVersionParsingFailure=False: All legacy kube types successfully parsed
NamespaceDeletionContentFailure=False: All content successfully deleted, may be waiting on finalization
NamespaceContentRemaining=True: Some resources are remaining: campaigns.bookstore.example.com has 1 resource instances
NamespaceFinalizersRemaining=True: Some content in the namespace has finalizers remaining: bookstore.example.com/storefront-cleanup in 1 resource instances

The namespace waits forever, and the conditions state the reason plainly: how many objects of which type remain, and which finalizer is waiting. The safe fix is to find the finalizer's owner and bring it back. I restarted the controller:

07:40:47 watching campaigns in namespace bookstore-promo
07:40:47 event Added winter-sale (generation 2)
07:40:47 bookstore-promo/winter-sale: banner removed from storefront, releasing finalizer
07:40:47 event Deleted winter-sale (generation 2)

$ kubectl get namespace bookstore-promo
Error from server (NotFound): namespaces "bookstore-promo" not found

Now the shortcut. On another campaign, with the controller stopped, I emptied the finalizers by hand:

$ kubectl -n bookstore delete campaign spring-sale --wait=false
campaign.bookstore.example.com "spring-sale" deleted from bookstore namespace
$ kubectl -n bookstore patch campaign spring-sale --type=merge -p '{"metadata":{"finalizers":null}}'
campaign.bookstore.example.com/spring-sale patched
$ kubectl -n bookstore get configmap campaign-spring-sale
Error from server (NotFound): configmaps "campaign-spring-sale" not found
$ cat storefront-banners.json; echo
{"bookstore/spring-sale":"Spring Sale"}

The Kubernetes side looks spotless, but the banner stayed in the storefront and there is no longer any record that would remove it. In real life this could be a cloud load balancer, a DNS record or a disk that keeps being billed. An unreachable APIService in the cluster produces a similar hang; then you see NamespaceDeletionDiscoveryFailure=True. The order is always the same: read the conditions, find the controller that owns the finalizer and run it; if that controller really is gone, clean up the external resource by hand and remove the finalizer last.

Choosing an Operator and Writing One in .NET

Installing an operator means adding a long-running program with broad permissions to your cluster. Look at three things when choosing one. Maturity: the project's maintainers, release cadence, CNCF status, and compatibility of its version with your Kubernetes version. The RBAC permissions it asks for: read the install manifest before applying it. In the static manifest of cert-manager v1.21.3:

$ kubectl create --dry-run=client -f cert-manager.yaml -o json | jq -s -r 'group_by(.kind) | map("\(.[0].kind)=\(length)") | join(" ")'
ClusterRole=13 ClusterRoleBinding=10 CustomResourceDefinition=6 Deployment=3 MutatingWebhookConfiguration=1 Namespace=1 Role=3 RoleBinding=3 Service=3 ServiceAccount=3 ValidatingWebhookConfiguration=1
$ kubectl create --dry-run=client -f cert-manager.yaml -o json | jq -r 'select(.kind == "ClusterRole") | .metadata.name as $n | .rules[] | select(.resources | index("secrets")) | "\($n): \(.verbs | join(","))"'
cert-manager-cainjector: get,list,watch
cert-manager-controller-issuers: get,list,watch,create,update,delete
cert-manager-controller-clusterissuers: get,list,watch,create,update,delete
cert-manager-controller-certificates: get,list,watch,create,update,delete,patch
cert-manager-controller-orders: get,list,watch
cert-manager-controller-challenges: get,list,watch
cert-manager-controller-challenges: get,list,watch

cert-manager's controller can create and delete Secrets in every namespace of the cluster; that is its job, but you need to know who holds this permission. Ask the same question for External Secrets Operator (the sixth post of the Configuration and Secret Management series) and Argo CD. The upgrade path: CRDs are versioned too, and Helm does not update the CRDs in a chart's crds/ folder on helm upgrade, nor delete them on uninstall; follow the upgrade steps in the operator's documentation.

For your own controller there are two options on the .NET side. KubernetesClient (the official client, 21.0.8 in October 2026) makes you write everything by hand, as in this post: watch, queue, retry, leader election. KubeOps is an operator SDK inspired by kubebuilder: it generates the CRD from an entity and gives you a ready structure for finalizers and reconcile; its latest stable release is 13.3.6, and the 14.0 prereleases target .NET 10 and 11. But first consider not writing one. A Job for a one-off task, a CronJob for periodic work, and a chart for a packaging need that Helm or Kustomize solves are enough. An operator makes sense when there is a state that has to be watched and corrected continuously; and the moment you write one, you also take on its upgrades, its RBAC and its outages.

Production Notes

When you put the controller into the cluster, give it only the permissions its job needs and test them with kubectl auth can-i:

$ kubectl auth can-i patch campaigns.bookstore.example.com --subresource=status -n bookstore --as=system:serviceaccount:bookstore:campaign-controller
yes
$ kubectl auth can-i delete configmaps -n bookstore --as=system:serviceaccount:bookstore:campaign-controller
no
$ kubectl auth can-i get secrets -n bookstore --as=system:serviceaccount:bookstore:campaign-controller
no

If you run more than one replica you need leader election, so that two controllers do not reconcile the same object at the same time. Reconcile must be idempotent: running ten times for the same object must give the same result as running once; the repeated Modified events in our log show why that is a must. If the external cleanup a finalizer does fails, do not remove the finalizer; retry, and make it visible with an event or a status condition. If the CRD schema needs a backward-incompatible change, add a new version (v1beta1) and plan the conversion between versions; do not break the existing version in place. In the eighteenth post we saw Kubernetes' own APIs move from v1beta1 to v1; the CRD versions of the operators you install age the same way, so check them before an upgrade too.

Common Pitfalls

Deleting a CRD deletes all of its custom resources. I deleted the CRD while the controller was stopped; one campaign had a finalizer, one did not:

$ kubectl delete crd campaigns.bookstore.example.com --wait=false
customresourcedefinition.apiextensions.k8s.io "campaigns.bookstore.example.com" deleted
$ kubectl -n bookstore get campaigns -o custom-columns=NAME:.metadata.name,FINALIZERS:.metadata.finalizers
NAME            FINALIZERS
classics-week   [bookstore.example.com/storefront-cleanup]
$ kubectl get crd campaigns.bookstore.example.com -o jsonpath='{.metadata.finalizers}{"\n"}{range .status.conditions[*]}{.type}={.status}: {.message}{"\n"}{end}'
["customresourcecleanup.apiextensions.k8s.io"]
NamesAccepted=True: no conflicts found
Established=True: the initial names have been accepted
Terminating=True: CustomResource deletion is in progress

The campaign without a finalizer went immediately; the other one, and with it the CRD itself, waited until the controller was restarted and did its cleanup. When removing a chart or cleaning up "unused" CRDs, this can mean deleting all of your production data. My first controller version taught a lesson in this experiment too: when the CRD went away, the watch got a 404 and the process crashed; that is why the catch (HttpOperationException) is in the code above.

The rest: force-removing a finalizer and leaving external resources orphaned. Pointing ownerReferences at an owner in another namespace; Kubernetes does not support cross-namespace ownership. Trying to write status through the same endpoint as spec (with the status subresource enabled, that change is ignored). Writing reconcile by event type ("create on Added") and drifting out of sync when events are missed. And writing an operator for every need.

A Small Experiment

★☆☆ Run kubectl get crd on your own cluster and find which operator each group in the list belongs to. Draw a Deployment's ownership tree with kubectl tree.

★★☆ Add status.conditions to the Campaign CRD and have the controller write a Ready=False condition when it cannot write to the storefront. Make the storefront file read-only and watch what happens during deletion.

★★★ Move the controller into the cluster as a Deployment: build its image, run it with this post's Role, add two replicas and leader election. Create campaigns while deleting one replica and check whether any events are lost.

With this post the extra posts of the Kubernetes series, and the series itself, are complete. On top of the road from pods to a managed cluster we added where pods land, using kubectl efficiently in daily work, and extending Kubernetes with your own types. The natural next stops are the site's Helm, GitOps and Argo CD, and Observability series: packaging the app, deploying it to the cluster from Git, and seeing what happens in production.