AI Infrastructure Planning: Compute, Kubernetes, GPUs and Capacity

AI workloads change infrastructure requirements because they have different resource profiles than traditional web applications. A web application is CPU-bound on request handling, memory-bound on session state, and network-bound on response size. An AI workload is GPU-bound on training and large-model inference, memory-bound on model weights and KV cache, and latency-bound on token generation rate. The bottleneck is not the application code but the model — a 70-billion-parameter model occupies 140 GB of GPU memory in float16 before a single token is generated, and the inference rate is bounded by memory bandwidth, not compute.

The implication is that capacity planning for AI cannot use the web-application playbook. A web application scales horizontally: add more CPU instances behind a load balancer, and throughput increases linearly until a network or database bottleneck appears. An LLM inference endpoint does not scale horizontally for free — each replica must load the full model into GPU memory, so adding a replica adds GPU memory cost, model loading time, and idle cost when the replica is not serving. Autoscaling an LLM endpoint is a decision about when to pay for another GPU replica, not a transparent horizontal scale. The planner must understand the model's memory footprint, loading time, and per-request cost before choosing an autoscaling policy.

AI workloads also introduce new failure modes. A GPU that runs out of memory (OOM) crashes the process, not just the request — a single request that exceeds the memory budget can bring down the replica. A model that takes 60 seconds to load means a cold-start adds a minute of latency, so autoscaling must keep warm capacity. A training job that fails after 10 hours of GPU time loses the work unless checkpoints are configured. These failure modes are infrastructure decisions: memory limits, warm capacity, checkpointing frequency. The capacity planner is not just sizing for throughput but for failure tolerance — how much capacity is needed to survive a failure without degrading the SLO.

The CPU-vs-GPU decision is determined by the model size, the latency SLO, and the cost. Small models — a scikit-learn classifier, a small embedding model, a fine-tuned DistilBERT — run efficiently on CPUs and do not benefit from GPU acceleration. A CPU is cheaper, more available, and easier to autoscale than a GPU, and for small models the latency is comparable. The threshold is roughly: if the model fits in CPU memory (a few gigabytes) and the inference latency on CPU meets the SLO, use a CPU. GPUs are justified when the model is too large for CPU memory, the latency on CPU exceeds the SLO, or the throughput on CPU is insufficient for the request rate.

LLM inference changes the calculus. A 7-billion-parameter model in float16 occupies 14 GB of GPU memory — it fits on a single consumer GPU but not in CPU memory at the same precision. Quantization (int8, int4) reduces the memory footprint and may allow a 7B model to run on a CPU, but at a latency cost that may break the SLO. The decision for LLMs is not CPU vs GPU but which GPU: the GPU's memory size determines which models fit, the GPU's memory bandwidth determines the token generation rate, and the GPU's cost determines the unit economics. A single A100 80GB can serve a 70B model in int8; a smaller GPU cannot. The planner must match the model to the GPU, not the GPU to the workload.

The mistake to avoid is using GPUs for workloads that do not need them. A GPU that serves a small model at low request rate is idle most of the time — the most expensive idle capacity in the infrastructure. A CPU instance serving the same model at the same latency costs a fraction of the GPU. The comparison table below maps workload types to CPU and GPU suitability, so the decision is based on the model and SLO, not on the assumption that AI means GPU. GPUs are a tool for specific workloads, not a default for all AI.

Training and inference have fundamentally different capacity profiles. Training is a batch workload: it runs for hours or days at full GPU utilisation, uses a fixed number of GPUs for the duration, and must checkpoint so that a failure does not lose all progress. Training capacity is sized for the job's duration — the planner provisions N GPUs for T hours, and the cost is N × T × GPU-hour-rate. Training does not need idle capacity or autoscaling; it needs checkpointing, fault tolerance, and the ability to resume. The capacity decision for training is: how many GPUs, for how long, at what cost, and what is the checkpoint interval.

Inference is a serving workload: it must handle variable request rates, meet a latency SLO, and keep warm capacity to avoid cold starts. Inference capacity is sized for peak load with a safety margin — the planner provisions enough GPU replicas to handle the peak request rate within the SLO, and keeps them warm. The cost is dominated by idle time: a replica that serves 10 requests per second at peak and 1 request per second at trough is 90% idle during the trough. The capacity decision for inference is: how many warm replicas, what autoscaling policy, and how to handle the burst without cold-start latency. Training and inference capacity are separate pools — training uses spot or preemptible GPUs, inference uses on-demand GPUs that stay warm.

The two workloads interact through model updates. A new model trained on training capacity must be deployed to inference capacity — the model registry is the handoff. The planner must account for the deployment: a model that is 140 GB takes time to download to each inference replica, and the rollout must be staged (canary, blue-green) to avoid a cold-start stampede where all replicas reload simultaneously and the serving SLO is violated during the rollout. Training and inference capacity planning are separate disciplines, but the deployment that connects them is a capacity event that must be planned, not improvised.

Kubernetes helps with AI workloads when the workloads need orchestration: scheduling GPU-enabled pods, managing job queues, autoscaling by queue depth or GPU utilisation, and handling preemption and eviction. Kubernetes provides workload controllers — Deployments for inference, Jobs and CronJobs for batch, and custom controllers for training — that manage replicas, restarts, and resource limits. For an organisation running multiple AI workloads, Kubernetes provides a common scheduling and resource-management layer that the platform team can centralise, so each team does not build its own scheduling infrastructure.

The key Kubernetes features for AI are GPU scheduling, resource quotas, and pod-level resource limits. GPU scheduling uses device plugins (NVIDIA GPU Operator) that expose GPUs as schedulable resources — a pod requests N GPUs and the scheduler places it on a node with available GPUs. Resource quotas prevent a team from consuming all GPUs — each namespace gets a quota, and the scheduler rejects pods that exceed it. Pod-level resource limits prevent a single request from OOMing a replica — the pod is killed and restarted, not the entire node. These features make Kubernetes a viable platform for multi-tenant AI workloads, where multiple teams share a GPU cluster.

Kubernetes does not help when the workload is a single model on a single GPU with no orchestration need. A single LLM endpoint on a dedicated GPU is simpler to run on a VM than on Kubernetes — the orchestration overhead (control plane, CNI, device plugin) adds complexity without value. Kubernetes helps at scale: multiple models, multiple teams, shared GPUs, autoscaling, preemption. The decision is whether the orchestration value exceeds the operational cost, and the threshold is similar to the platform threshold — a handful of workloads can run on VMs, a dozen or more benefit from Kubernetes.

Concurrency is the number of in-flight requests a single replica can handle. For an LLM inference endpoint, concurrency is bounded by GPU memory: each in-flight request occupies KV cache memory, and when the KV cache is full, new requests queue. The planner must size the concurrency per replica — how many requests can be in flight simultaneously — and then size the number of replicas to handle the peak request rate: replicas = peak_request_rate / per_replica_concurrency. If each replica handles 10 concurrent requests and the peak is 100 requests per second with 100 ms latency, the system needs 10 replicas. The math is simple; the measurement of per-replica concurrency is not — it depends on the model, the sequence length, and the KV cache size, and must be measured, not assumed.

Queueing absorbs bursts. A queue in front of the inference endpoint allows the system to handle request spikes that exceed capacity without rejecting requests — the queue holds the excess, and the replicas process it as capacity becomes available. The trade-off is latency: a request in the queue waits for a replica, adding queue latency to inference latency. The SLO must account for queue latency: if the SLO is 500 ms and inference takes 200 ms, the queue budget is 300 ms — beyond that, requests should be rejected rather than queued. The queue depth is a capacity signal: a growing queue triggers autoscaling, and a queue that exceeds the latency budget triggers load shedding.

Autoscaling for AI workloads is driven by different signals than web autoscaling. Web autoscaling uses CPU utilisation; AI autoscaling must use GPU utilisation, queue depth (for async), or consumer lag (for streaming). The autoscaler must account for cold-start time: a new GPU replica takes 30–60 seconds to load the model, so the autoscaler must scale ahead of demand, not behind it. Predictive autoscaling — scaling based on forecast traffic, not current load — reduces cold-start risk for workloads with predictable patterns (business-hours traffic). The autoscaling policy must be tested under realistic load: an autoscaler that has never been load-tested will scale too slowly or too aggressively, both of which cause SLO violations.

Storage for AI workloads has three components: model weights, training data, and inference state (KV cache, agent checkpoints). Model weights are large (tens of GB), read once at replica startup, and must be available at high bandwidth so that cold starts are fast. The storage design is: a high-bandwidth, read-optimised store (a distributed filesystem or object store with a local cache) that serves model weights to replicas on demand. A cold start that downloads 140 GB over a 1 Gbps link takes 18 minutes — unacceptable. The storage must be co-located with the GPUs (local NVMe or a high-bandwidth network filesystem) so that model loading takes seconds, not minutes.

Training data storage must handle high-throughput reads and large datasets. A training job that reads a 1 TB dataset over a network filesystem needs a storage system that can sustain the read bandwidth — typically 1–10 Gbps per worker — without becoming the bottleneck. The design is: a parallel filesystem (Lustre, GPFS) or an object store with a local cache, with data partitioned so that workers read different shards in parallel. Storage I/O is often the hidden bottleneck in training: a GPU that waits for data is an idle GPU, and the cost is the same as a busy GPU. The planner must measure read bandwidth, not just GPU utilisation.

Networking for AI workloads must handle two patterns: east-west traffic between replicas (for distributed training, where GPUs exchange gradients) and north-south traffic to clients (for inference). Distributed training is network-intensive: a multi-GPU training job uses NCCL (NVIDIA Collective Communications Library) over high-bandwidth interconnects (NVLink, InfiniBand) to exchange gradients, and the network bandwidth determines the training efficiency. Inference networking is lighter — each request is small — but the latency must be low. The network design is: high-bandwidth, low-latency interconnect for training; standard networking for inference. Mixing training and inference on the same network can cause contention — the training gradient exchange saturates the network and the inference SLO degrades.

Availability for AI workloads is planned across failure domains: a single GPU failure, a node failure, a rack failure, or an availability-zone failure. Each failure domain has a probability and a blast radius. A single GPU failure is common (GPUs fail more often than CPUs) and takes down one replica — the load balancer routes around it. A node failure takes down all replicas on that node — if all replicas are on one node, the workload is down. A zone failure takes down all nodes in that zone — if all nodes are in one zone, the workload is down. The availability design spreads replicas across failure domains so that no single failure takes down the workload.

The SLO determines the required availability. A 99.9% SLO allows 43 minutes of downtime per month; a 99.99% SLO allows 4 minutes. For AI workloads, the availability calculation must include cold-start time: a workload that loses a replica and starts a new one has a cold-start gap (30–60 seconds) during which the capacity is reduced. If the SLO is 99.99% and the cold-start is 60 seconds, the workload cannot tolerate more than 3 cold starts per month — a tight constraint. The planner must either keep spare warm capacity (reducing cold starts at the cost of idle capacity) or accept the SLO risk. The availability decision is a trade-off between idle capacity cost and SLO risk, and it must be explicit, not implicit.

Failure-domain planning for training is different. A training job that runs for 10 hours on 8 GPUs has a meaningful probability of a GPU failure during the run — if the failure is not handled, the job restarts from the beginning. The design is checkpointing: the job saves its state to durable storage every N minutes, and on failure, it resumes from the last checkpoint. The checkpoint interval is a trade-off: frequent checkpoints reduce lost work but add overhead; infrequent checkpoints save overhead but lose more work on failure. The Kubernetes Job controller with backoffLimit and the training framework's checkpointing (PyTorch Lightning, HuggingFace Trainer) work together: the Job restarts the pod, the checkpoint is loaded, and training resumes. The planner must configure both, not assume one handles it.

Cloud infrastructure provides GPUs on demand, with no capital expenditure and the ability to scale up and down. The cost is per GPU-hour, which is higher than owning GPUs but has no idle cost when the workload stops. Cloud is the right choice for workloads that are bursty (need GPUs for a few hours, then none), experimental (the team is not sure the model will work), or early-stage (the workload has not stabilised). The trade-off is unit economics: at sustained utilisation (GPUs running 24/7), cloud GPU-hour cost exceeds the cost of owning GPUs within 12–18 months. The crossover point depends on the GPU type, the cloud provider, and the utilisation.

Self-hosted infrastructure (owned GPUs in a data centre) has a high capital cost but a low marginal cost — the GPU is paid for, and the only ongoing cost is power and cooling. Self-hosted is the right choice for workloads with sustained, high utilisation: a training cluster that runs jobs continuously, an inference endpoint with steady traffic that keeps GPUs busy 24/7. The trade-off is capacity: a self-hosted cluster cannot grow instantly — adding GPUs takes weeks (procurement, installation) — so the cluster must be sized for peak, which means idle capacity during troughs. The decision is whether the sustained utilisation justifies the capital cost and the capacity rigidity.

Hybrid infrastructure combines cloud and self-hosted: a self-hosted baseline for sustained workloads, cloud burst for peaks. The design is: run the steady-state inference on owned GPUs, burst to cloud GPUs when traffic exceeds the owned capacity. The hybrid pattern is operationally complex — the workloads must be portable (containerised, with the model in a registry that both environments can access), and the cost model must account for the cloud burst cost. Hybrid is the right choice when the workload has a predictable baseline and occasional peaks that do not justify owning peak capacity. The planner must measure the baseline, the peak frequency, and the burst cost to decide whether hybrid saves money over all-cloud or all-self-hosted.

Unit economics for AI is the cost per unit of value: cost per request, per token, per inference, per agent run, per outcome. Total cloud spend is not a useful metric because it does not connect cost to value — a system that costs $10,000/month and serves 1 million requests has a unit cost of $0.01 per request; a system that costs $5,000/month and serves 100,000 requests has a unit cost of $0.05 per request. The cheaper system on the total bill is more expensive per unit of value. The planner must define the unit of value and measure cost per unit, not total cost.

The unit of value depends on the workload. For an LLM endpoint, the unit is the token: cost per 1M tokens (input + output). For a classification endpoint, the unit is the request: cost per request (GPU time + infrastructure). For an agent workflow, the unit is the run: cost per completed workflow (tokens + tool calls + compute). For a business outcome, the unit is the outcome: cost per resolved ticket, per generated report, per successful recommendation. The unit economics must be connected to the business metric — cost per outcome — not just the infrastructure metric — cost per request. A system that optimises cost per request but degrades the outcome quality (more requests needed per resolution) is more expensive per outcome, not cheaper.

The unit economics drive the optimisation decisions. If cost per token is dominated by GPU idle time, the lever is autoscaling or request batching. If cost per token is dominated by model size, the lever is quantization or a smaller model. If cost per outcome is dominated by retry rate (the model produces bad outputs that must be re-run), the lever is quality improvement, not cost reduction. The planner must measure the unit economics, identify the dominant cost driver, and apply the lever that addresses it. Cost optimisation without unit economics is guesswork — a team that reduces total spend without measuring per-unit cost may be reducing value, not cost.