Scaling in Kubernetes: HPA, VPA, In-Place Resize, KEDA and PodDisruptionBudgets

Version note: The examples in this post were run with kind 0.33 and Kubernetes 1.37, kubectl 1.37, metrics-server 0.9.1, KEDA (Kubernetes Event-driven Autoscaling) 2.21, RabbitMQ 4.4 and .NET 10.

Guess First

A Bookstore API (Application Programming Interface) pod runs with limits.memory: 256Mi. On a campaign day memory runs short, and you raise the limit to 512Mi without restarting the pod, using in-place resize, which reached GA (General Availability) in Kubernetes 1.35. Kubernetes says "resize completed" and the restart count is still 0. Question: does the GC (Garbage Collector) of the .NET runtime inside now work against 512Mi, or still against 256Mi? Second question: if you attach a CPU-based (Central Processing Unit) HPA (Horizontal Pod Autoscaler) to a Deployment that has no CPU request, what happens: does the pod count never grow, or grow forever? The answers are in the post, with real cluster output.

What HPA Measures: metrics-server and Percent of Request

HPA is a controller: at a fixed interval (15 seconds by default) it reads the metrics of the target Deployment's pods, computes the desired replica count and updates the Deployment's scale subresource. CPU and memory metrics require metrics-server in the cluster; managed clusters usually ship it, on kind we install it ourselves. Because kind's kubelet certificates are not signed by the cluster CA (Certificate Authority), we ran metrics-server with --kubelet-insecure-tls, in the test environment only:

kubectl apply -f metrics-server.yaml
kubectl top pods -n bookstore

NAME                             CPU(cores)   MEMORY(bytes)
api-79544c7fc7-m95j8             4m           45Mi
api-79544c7fc7-rqdgf             3m           52Mi

The "percent" HPA talks about is not the node's CPU but the container's requests.cpu. A pod with a 100m request using 50m is at 50%. That is why no percentage can be computed for a pod without a request. We deployed the same API as a Deployment without requests (api-norequest), attached an HPA to it as well, and kubectl get hpa showed:

NAME            REFERENCE                  TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
api             Deployment/api             cpu: 3%/60%          2         6         2          45s
api-norequest   Deployment/api-norequest   cpu: <unknown>/60%   2         6         2          45s

describe spells out the reason:

ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu in container api-norequest of Pod api-norequest-5d9865c7b5-h7jd5

So the answer to the second question: the HPA neither grows nor shrinks, it freezes at minReplicas and says so only in events. A team that sees "we have an HPA" on a dashboard and relaxes is left with two pods when the load arrives. How to pick request values was the topic of the ninth post in this series (Resource Requests and Limits); HPA only works as well as those values are realistic.

HPA v2 and behavior: How Fast to Grow, How Slowly to Shrink

The autoscaling/v2 HPA supports multiple metrics and lets you describe how fast to scale up and down. The HPA we wrote for the Bookstore API:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
  namespace: bookstore
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 6
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Pods
          value: 2
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 120
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60

The math is simple: desiredReplicas = ceil(currentReplicas × currentMetric / targetMetric). If the ratio is within 10% of 1 (the default tolerance), nothing happens. behavior tames that raw result: no stabilization on scale-up, but at most 2 pods added per 30 seconds; on scale-down the highest recommendation of the last 120 seconds wins and at most 1 pod is removed per minute. We generated load from a busybox pod with four parallel loops calling /api/diag/burn?seconds=1. We prefixed the output of kubectl get hpa api -w with the time:

06:28:45 api    Deployment/api   cpu: 3%/60%      2         6         2          60s
06:29:00 api    Deployment/api   cpu: 135%/60%    2         6         2          75s
06:29:15 api    Deployment/api   cpu: 1914%/60%   2         6         4          90s
06:29:45 api    Deployment/api   cpu: 814%/60%    2         6         6          2m
06:31:00 api    Deployment/api   cpu: 1%/60%      2         6         6          3m15s
06:33:01 api    Deployment/api   cpu: 1%/60%      2         6         5          5m16s
06:34:01 api    Deployment/api   cpu: 1%/60%      2         6         4          6m16s
06:35:01 api    Deployment/api   cpu: 1%/60%      2         6         3          7m16s
06:36:01 api    Deployment/api   cpu: 1%/60%      2         6         2          8m16s

Notice three things. 1914% is not a bug: the container has no CPU limit, its request is 100m, and the pod can use two idle cores on the node. The raw formula would have asked for 2 × 1914 / 60 ≈ 64 pods; the policy added 2 at a time and maxReplicas stopped it at 6. The load generator stopped at 06:30:24, the metric fell to 1% at 06:31:00, but the first scale-down came at 06:33:01: the 120-second window. Then one pod per minute. Not deleting pods right after a sudden drop means no cold start when a second wave arrives a few minutes later. Since cold start in .NET apps (JIT, connection pools, the EF (Entity Framework) Core model) takes a few seconds, keeping scale-down slow is usually the right call.

One more detail went GA in Kubernetes 1.37: behavior.scaleUp.tolerance and behavior.scaleDown.tolerance. They override the cluster-wide 10% tolerance per HPA; on a 50-pod Deployment, 10% means ignoring a five-pod gap. kubectl explain hpa.spec.behavior.scaleUp.tolerance shows the field's description: if unset, "the default cluster-wide tolerance is applied (by default 10%)".

VPA and In-Place Resize: Growing a Pod Vertically, Not Horizontally

HPA changes the number of pods; VPA (Vertical Pod Autoscaler) changes a pod's requests and limits. VPA is a separate project installed into the cluster (in the kubernetes/autoscaler repository, version 1.8.1 as of October 2026). It watches historical usage, produces recommendations and applies them according to updateMode: Off only writes recommendations, Initial applies them only to new pods, Recreate evicts and recreates the pod, and InPlaceOrRecreate (GA in VPA 1.6) first tries to grow the pod in place. Auto is deprecated and today behaves like Recreate. Running both HPA and VPA on CPU for the same Deployment is a classic conflict: one adds pods, the other grows requests, and each changes the other's metric.

The core feature that lets VPA resize in place is in-place pod resize. It arrived as beta, enabled by default, in Kubernetes 1.33 and went GA in 1.35. The resources in a pod spec can now be changed through the resize subresource:

apiVersion: v1
kind: Pod
metadata:
  name: api-resize
  namespace: bookstore
spec:
  containers:
    - name: api
      image: bookstore-api:1.2.0
      ports:
        - name: http
          containerPort: 8080
      resizePolicy:
        - resourceName: cpu
          restartPolicy: NotRequired
        - resourceName: memory
          restartPolicy: NotRequired
      resources:
        requests:
          cpu: 500m
          memory: 256Mi
        limits:
          cpu: "1"
          memory: 256Mi

kubectl -n bookstore patch pod api-resize --subresource resize --patch \
  '{"spec":{"containers":[{"name":"api","resources":{"requests":{"cpu":"1","memory":"512Mi"},"limits":{"cpu":"2","memory":"512Mi"}}}]}}'

Normal  ResizeStarted    5s    kubelet  Pod resize started: {"containers":[{"name":"api","resources":{"limits":{"cpu":"2","memory":"512Mi"},"requests":{"cpu":"1","memory":"512Mi"}}}],"generation":2}
Normal  ResizeCompleted  5s    kubelet  Pod resize completed: {"containers":[{"name":"api","resources":{"limits":{"cpu":"2","memory":"512Mi"},"requests":{"cpu":"1","memory":"512Mi"}}}],"generation":2}

Restart count 0. Now back to the opening question. The Bookstore API's /api/diag/runtime endpoint returns Environment.ProcessorCount and GC.GetGCMemoryInfo().TotalAvailableMemoryBytes. Before and after the resize:

{"processorCount":1,"isServerGC":false,"totalAvailableMemoryMB":192,"heapSizeMB":0,"gcHeapHardLimitMB":0,"workingSetMB":67}
{"processorCount":1,"isServerGC":false,"totalAvailableMemoryMB":192,"heapSizeMB":0,"gcHeapHardLimitMB":0,"workingSetMB":68}

Kubernetes raised the cgroup limit to 512Mi, but .NET still sees 192 MB (75% of 256Mi, the default heap hard limit from the ninth post). The runtime reads these values when the process starts; even though the CPU limit went up to 2 cores, ProcessorCount also stayed at 1. GC.RefreshMemoryLimit(), introduced in .NET 8, makes the GC re-read the limits. We added a small endpoint that calls it:

// re-reads the cgroup memory limit after an in-place resize (.NET 8+)
app.MapPost("/api/diag/refresh-memory-limit", () =>
{
    long before = GC.GetGCMemoryInfo().TotalAvailableMemoryBytes / 1024 / 1024;
    GC.RefreshMemoryLimit();
    long after = GC.GetGCMemoryInfo().TotalAvailableMemoryBytes / 1024 / 1024;
    return Results.Ok(new { beforeMB = before, afterMB = after, processorCount = Environment.ProcessorCount });
});

{"beforeMB":192,"afterMB":384,"processorCount":1}

Memory is fixed, ProcessorCount is still 1: decisions that depend on the core count, such as the thread pool and the number of Server GC heaps, stay fixed for the life of the process. The practical takeaway: for a memory resize to actually help a .NET service, either the app calls RefreshMemoryLimit or resizePolicy uses RestartContainer for memory. We tried the second; the container restarted (RESTARTS 1) and the new process saw 384 MB:

NAME          READY   STATUS    RESTARTS     AGE
api-resize2   1/1     Running   1 (8s ago)   8s
{"processorCount":1,"isServerGC":false,"totalAvailableMemoryMB":384,"heapSizeMB":0,"gcHeapHardLimitMB":0,"workingSetMB":67}

A container restart is cheaper than a pod restart: the pod stays on the same node with the same IP (Internet Protocol) address, with no scheduling and no image pull. We also saw both limits of resize in real output. A resize that would change the QoS (Quality of Service) class is rejected (Pod QOS Class may not change as a result of resizing), and a request exceeding the node's capacity was refused immediately by the API server on our 1.37 cluster: node didn't have enough allocatable resources: cpu, requested: 32000, allocatable: 10000.

KEDA: Scaling Workers by Queue Length and Scale-to-Zero

CPU is a good signal for an HTTP (Hypertext Transfer Protocol) API; it is not for a worker that consumes a queue. Bookstore.Worker, which sends order confirmation e-mails, spends most of its time waiting on SMTP (Simple Mail Transfer Protocol), so its CPU stays low while the queue grows. KEDA fills that gap: it has scalers that read external sources such as queues, streams and cron, and behind the scenes it creates an HPA it manages. The step between 0 and 1 it handles itself instead of leaving it to the HPA. The worker's consuming code is the same as in the RabbitMQ.Client 7 post of the Working with Message Queues series; the two lines that matter here are prefetchCount: 1 and the manual ack:

public sealed class OrderEmailConsumer(IConfiguration configuration, ILogger<OrderEmailConsumer> logger)
    : BackgroundService
{
    private const string QueueName = "order-emails";

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        ConnectionFactory factory = new() { Uri = new Uri(configuration["RabbitMq:Uri"]!) };
        await using IConnection connection = await factory.CreateConnectionAsync(stoppingToken);
        await using IChannel channel = await connection.CreateChannelAsync(cancellationToken: stoppingToken);

        await channel.QueueDeclareAsync(QueueName, durable: true, exclusive: false, autoDelete: false,
            cancellationToken: stoppingToken);
        await channel.BasicQosAsync(prefetchSize: 0, prefetchCount: 1, global: false, stoppingToken);

        AsyncEventingBasicConsumer consumer = new(channel);
        consumer.ReceivedAsync += async (_, ea) =>
        {
            string orderId = Encoding.UTF8.GetString(ea.Body.Span);
            // simulates rendering and sending one order confirmation e-mail
            await Task.Delay(TimeSpan.FromMilliseconds(500), stoppingToken);
            await channel.BasicAckAsync(ea.DeliveryTag, multiple: false, stoppingToken);
            logger.LogInformation("Sent confirmation e-mail for order {OrderId}", orderId);
        };

        await channel.BasicConsumeAsync(QueueName, autoAck: false, consumer, stoppingToken);
        logger.LogInformation("Consuming {Queue} on {Host}", QueueName, Environment.MachineName);
        await Task.Delay(Timeout.Infinite, stoppingToken).ContinueWith(_ => { }, TaskScheduler.Default);
    }
}

We installed KEDA with Helm (helm install keda kedacore/keda --version 2.21.0 -n keda --create-namespace; Helm itself is the topic of the upcoming Helm series). KEDA 2.21's official test matrix lists Kubernetes 1.34–1.36; the scenario in this post worked fine on 1.37, but in production don't upgrade without checking the compatibility table. The worker Deployment starts with replicas: 0 and KEDA decides the scale:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: rabbitmq
  namespace: bookstore
spec:
  secretTargetRef:
    - parameter: host
      name: rabbitmq
      key: uri
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker
  namespace: bookstore
spec:
  scaleTargetRef:
    name: worker
  minReplicaCount: 0
  maxReplicaCount: 5
  pollingInterval: 5
  cooldownPeriod: 30
  triggers:
    - type: rabbitmq
      metadata:
        protocol: amqp
        queueName: order-emails
        mode: QueueLength
        value: "20"
      authenticationRef:
        name: rabbitmq

value: "20" means "20 messages per pod". The connection address comes from a Secret; an AMQP (Advanced Message Queuing Protocol) URI (Uniform Resource Identifier) contains a user name and password, so it doesn't go into the manifest. We published 300 messages to the queue:

06:33:45 keda-hpa-worker   Deployment/worker   <unknown>/20 (avg)   1         5         0          24s
06:33:52 keda-hpa-worker   Deployment/worker   300/20 (avg)         1         5         1          31s
06:34:07 keda-hpa-worker   Deployment/worker   75250m/20 (avg)      1         5         4          46s
06:34:22 keda-hpa-worker   Deployment/worker   55800m/20 (avg)      1         5         5          61s
06:34:52 keda-hpa-worker   Deployment/worker   0/20 (avg)           1         5         5          91s
06:35:22 keda-hpa-worker   Deployment/worker   <unknown>/20 (avg)   1         5         0          2m1s

Normal  KEDAScaleTargetActivated    2m36s  keda-operator  Scaled apps/v1.Deployment bookstore/worker from 0 to 1, triggered by s0-rabbitmq-order-emails
Normal  KEDAScaleTargetDeactivated  67s    keda-operator  Deactivated apps/v1.Deployment bookstore/worker from 5 to 0

KEDA took the Deployment from 0 to 1 itself, the HPA carried it from 1 to 5 (75250m means 75.25 messages per pod). After the queue drained and cooldownPeriod elapsed, the Deployment went back to 0. The price of scale-to-zero is the latency of the first message: scheduling the pod, pulling the image (if missing) and starting the .NET host. In our run the worker image had not been loaded onto the nodes, so the first pods fell into ImagePullBackOff; KEDA scaled correctly, but the messages were processed only after the image was loaded. An order e-mail can be a few seconds late; if a payment confirmation can't, keep minReplicaCount: 1. In Kubernetes 1.37 the HPA's own scale-to-zero support (HPAScaleToZero) became beta and enabled by default, but it only works with Object and External metrics and you still need an adapter to provide the external metric; for the queue scenario KEDA is still the shortest path.

PodDisruptionBudget and Topology Spread: Staying Up While Scaling

Adding pods only helps as long as they don't pile up on the same node. topologySpreadConstraints spreads pods evenly across nodes (or zones). Our first attempt got stuck on kind in an unexpected way:

Warning  FailedScheduling  2m7s  default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match pod topology spread constraints. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.

The reason: nodeTaintsPolicy defaults to Ignore. The tainted control-plane node also counts as a topology domain, and it has 0 pods; putting a third pod on either worker makes the skew 2, which maxSkew: 1 forbids. During a rolling update the pods of the old and new ReplicaSets are counted together, so the new pod found no place at all. The fixed version:

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          nodeTaintsPolicy: Honor
          matchLabelKeys:
            - pod-template-hash
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: api

matchLabelKeys: [pod-template-hash] counts only the pods of the same revision. In production the topology key is usually topology.kubernetes.io/zone; the goal is for half of the service to survive the loss of a zone.

A PDB (PodDisruptionBudget) limits voluntary disruptions: operations that go through the eviction API, such as kubectl drain, cluster upgrades and the cluster autoscaler removing a node. A pod crashing or a node dying on its own ignores the PDB. In the first attempt we wrote minAvailable: 2 while the replica count was 2:

evicting pod bookstore/api-6c84998f86-9wqzq
error when evicting pods/"api-6c84998f86-9wqzq" -n "bookstore" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
...
error: unable to drain node "k8swork2-1-worker" due to error: error when evicting pods/"api-6c84998f86-9wqzq" -n "bookstore": global timeout reached: 30s, continuing command...

A minAvailable equal to the replica count means "no pod may leave voluntarily"; a managed cluster's node upgrade either hangs for hours or the provider ignores the PDB after some time. The right choice is maxUnavailable: 1:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
  namespace: bookstore
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: api

NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
api    N/A             1                 1                     3m19s
evicting pod bookstore/api-6c84998f86-9wqzq
pod/api-6c84998f86-9wqzq evicted
node/k8swork2-1-worker drained

maxUnavailable behaves better for Deployments that grow with an HPA: it allows one pod whether there are 2 replicas or 6. The drain also stopped once with cannot delete Pods that declare no controller: bare pods without a controller (the curl pod we started for testing) block a drain. There should be no controller-less pods in production.

When Nodes Run Out: Cluster Autoscaler and Karpenter

If there is no room on a node for the pod the HPA adds, the pod stays Pending. Adding nodes is a cluster-level job. Cluster Autoscaler sees Pending pods and grows the cloud provider's node group: a node pool on AKS (Azure Kubernetes Service) and GKE (Google Kubernetes Engine), a managed node group on EKS (Amazon Elastic Kubernetes Service); it also drains and removes nodes that stay empty. Karpenter (which came from AWS, that is Amazon Web Services, and is now also available on AKS as "node auto provisioning") does not need predefined node groups; it picks a suitable VM (Virtual Machine) type for the pods' needs and launches it directly. Both use eviction when removing nodes, so they respect PDBs; an app whose PDB never allows a disruption blocks node removal and therefore cost savings. These tools don't run on kind; we'll talk about moving to a managed cluster in the seventeenth post of the series.

Production Notes

The scaling chain is only as fast as its weakest link: metrics-server measures every 15 seconds, HPA decides every 15 seconds, the new pod is scheduled, the image is pulled, the .NET app starts, the readiness probe passes. In our measurement the first scaling decision came about 30 seconds after the load started. If traffic spikes within seconds, HPA won't keep up; keep minReplicas high enough to carry normal traffic and treat HPA as a tool for gradual growth, not for sudden peaks. For new pods to be ready fast, keep the image small and tie readiness to the moment the app is really ready (the eighth post of the series). Choose a CPU target around 60–70%: a 90% target chokes the existing pods until new ones arrive. To measure, kubectl describe hpa events and the HPA's own status are a sufficient start; seeing replica count, queue depth and p95 latency on one graph is the topic of the upcoming Observability series. Finally, if an HPA manages a Deployment, don't pin replicas in the manifest; every kubectl apply overrides the HPA's decision and pulls the replica count back. Removing the field from the manifest is the cleanest way.

Common Pitfalls

Attaching a CPU-based HPA to a pod without requests: an <unknown> target and zero scaling. Building an HPA on memory: in .NET the GC doesn't return memory right away, so memory usage may not drop after the load goes away and the HPA never shrinks; use a memory metric only if it really tracks the workload. Assuming .NET sees the new limit after an in-place resize: the output above shows it doesn't. Running HPA and VPA together on the same CPU metric. Setting minAvailable equal to the replica count and locking node upgrades. Choosing minReplicaCount: 0 in KEDA without accounting for the latency of the first message. Forgetting that topology spread counts the control-plane node or cordoned nodes as domains. And finally, scale-down cutting long requests: a pod removed by the HPA gets SIGTERM (the Unix termination signal) too, and if graceful shutdown (the tenth post of the series) isn't set up right, every scale-down produces a handful of 502s.

A Small Experiment

★☆☆ Set the HPA's scaleDown.stabilizationWindowSeconds to 0 and toggle the load on and off for 30 seconds. Compare how the replica count fluctuates in the kubectl get hpa -w output with the output above.

★★☆ Run the Bookstore API pod with limits.memory: 256Mi, fill memory with POST /api/diag/alloc-managed?mb=150, then resize the pod to 512Mi and try a second 150 MB allocation before and after calling RefreshMemoryLimit. In which case do you see OutOfMemoryException?

★★★ Add a cron scaler as a second trigger to the KEDA ScaledObject: keep at least one worker up on weekdays between 09:00 and 18:00, and let it drop to zero at night if the queue is empty. Watch which trigger is decisive at which moment with kubectl describe hpa keda-hpa-worker.

All the pods we scaled were stateless: deleting any of them lost no data. In the next post we move to the hard part: apps that keep state with volumes, PersistentVolumeClaims and StatefulSets, and the question "should we put the database in the cluster?"