AI System Design Course for Production AI Systems

01Requirements, Quality Attributes and Capacity

• ·Turn a use case into functional requirements and non-functional requirements (NFRs), including latency, availability, privacy and cost limits.
• ·Estimate queries per second (QPS), active requests, document growth, storage and network demand from explicit workload assumptions.
• ·Separate average behavior from p95 and p99 latency. Budget time across retrieval, model execution, tools and response delivery.
• ·Account for input and output tokens, traffic bursts and workload mix before choosing infrastructure.

Design output: A requirements brief and capacity estimate with assumptions that another engineer can challenge.

02Distributed Systems for AI Workloads

• ·Compare synchronous APIs, RPC, queues and publish/subscribe for interactive requests and background work.
• ·Design retries, timeout budgets, idempotency and deduplication around operations that may complete even when a response is lost.
• ·Use load balancing, partitioning, replication and caching with an explicit consistency and tenant-isolation model.
• ·Plan backpressure, dead-letter handling and graceful degradation so one slow dependency does not exhaust the service.

Design output: A service and failure-boundary diagram, including what happens when a dependency becomes slow or unavailable.

03Application, Gateway and Context Architecture

• ·Trace a request through identity checks, the application API, model routing, context construction and response handling.
• ·Compare provider abstraction with provider-specific capabilities, quotas, output formats and fallback behavior.
• ·Separate conversation state, durable records, caches and versioned prompt configuration. Define retention and context limits.
• ·Handle streaming, cancellation, invalid structured outputs and partial responses without treating a successful HTTP response as a successful task.

Design output: A request sequence and state model covering normal operation, cancellation and partial failure.

04RAG and Data System Design

• ·Design batch and incremental ingestion, document processing, versioned indexes and deletion propagation.
• ·Compare lexical, vector and hybrid retrieval; decide where chunking, reranking and metadata filtering belong.
• ·Enforce access rules before retrieved content enters the model context. Keep tenant and permission boundaries intact in retrieval and caches.
• ·Evaluate freshness, retrieval relevance, citation support and answer quality separately; choose what to measure offline and in production.

Design output: A retrieval architecture that explains data freshness, access enforcement and quality measurement.

For implementation depth in retrieval, model adaptation and evaluation, explore the Generative AI Course .

05Agentic AI System Design

• ·Decide when a deterministic workflow is sufficient and when an agent needs room to choose actions.
• ·Separate working state, long-term memory and the durable record of execution. Define checkpoints, resumption and safe handling of repeated events.
• ·Define tool boundaries, execution identities, permissions and human approval. Use the Model Context Protocol (MCP) as an integration protocol alongside explicit authorization.
• ·Compare a single agent with multiple agents through coordination cost, failure recovery, auditability and evaluation. Set time, step and spending limits.

Design output: An execution design showing who can act, what is persisted and how an interrupted workflow resumes without blindly repeating side effects.

For hands-on work with orchestration, MCP, memory and tool controls, explore the Agentic AI Course .

06Inference and Infrastructure Architecture

• ·Compare hosted model APIs with self-hosted models using privacy, control, workload, staffing and utilization assumptions.
• ·Reason about prefill, decoding, continuous batching, KV-cache capacity and prefix caching. Examine serving approaches such as vLLM and SGLang without turning the module into an engine installation guide.
• ·Separate time to first token, token delivery speed and complete response latency. Relate them to context length, concurrency and user experience.
• ·Plan quotas, admission control, warm capacity, CPU/GPU resources and autoscaling. Evaluate managed hosting and Kubernetes against the operating work they require.

Design output: An inference plan with measurable latency and throughput targets, a capacity hypothesis and an overload policy.

For deeper serving and LLM production operations, explore the LLMOps Course .

07Reliability, Security and Evaluation

• ·Define service-level indicators and objectives, error budgets, recovery time and recovery point objectives for the parts of the system that need them.
• ·Combine logs, metrics and traces with retrieval evidence, agent trajectories, token usage and cost. Control sensitive data in telemetry.
• ·Build evaluation sets and release checks for grounding, task completion, retrieval and tool use. Account for the limitations of automated judges and the need for human review.
• ·Threat-model prompt injection, poisoned content, unauthorized tools and data leakage. Combine least privilege, isolation, validation and approval controls with incident response and rollback plans.

Design output: A reliability and threat-model worksheet with detection signals, containment actions and release checks.

08Architecture Cases and Design Defence

• ·Apply the earlier modules to an enterprise RAG platform, a customer-support agent, a high-volume LLM API and a multimodal document workflow.
• ·Extend the discussion to a coding assistant and a reusable agent workflow platform, where repository access, tool execution and shared runtime controls change the design.
• ·Compare alternatives using workload estimates, failure analysis, security boundaries and operating cost.
• ·Record major choices in architecture decision records (ADRs), then revise the design when a requirement or assumption changes.

Design output: A coherent architecture pack that connects requirements to decisions, evidence and unresolved risks.

If your immediate gap is ML pipelines, registries and deployment automation, explore the MLOps Course .

Enterprise RAG with Changing Permissions

Problem

Employees need answers from a large document collection while document access and content change throughout the day.

Exercise assumptions

Two million documents, peak demand of 100 questions per second and a proposed indexing-freshness target of 15 minutes.

Decisions

Separate ingestion from serving, choose a retrieval and reranking strategy, version documents and enforce access restrictions before building context.

Failure to examine

A revoked permission or deleted document remains in an index or cache and is returned to an unauthorized user.

Trade-off

Faster updates require more processing and coordination; cached answers need an authorization and invalidation strategy.

A Support Agent That Can Take Action

Problem

An assistant answers support questions and can request an order change through business APIs.

Exercise assumptions

Twenty thousand conversations a day, bursty demand and actions that require approval above a defined risk threshold.

Decisions

Separate advice from execution, persist workflow state, validate tool arguments and tie approvals to a specific proposed action.

Failure to examine

A timeout triggers a retry after the order API has already completed the change.

Trade-off

More autonomy may reduce handling time, but it also increases the work needed for permissions, evaluation and recovery.

A Shared LLM API Under Burst Traffic

Problem

Several applications share a model gateway with different latency, quota and privacy requirements.

Exercise assumptions

In a stable workload averaging 200 requests per second and eight seconds from admission to completion, about 1,600 admitted requests are in progress at once. This estimates concurrency, not GPU capacity; token lengths, queuing and traffic bursts need their own analysis.

Decisions

Allocate tenant quotas, define admission control, compare serving options and budget warm capacity using token distributions as well as request counts.

Failure to examine

Long generations exhaust capacity and delay short interactive requests.

Trade-off

Higher utilization can improve unit cost while worsening tail latency and overload recovery.

A Multimodal Document Processing Service

Problem

A service extracts information from documents containing text, scanned pages and images, with a review path for uncertain results.

Exercise assumptions

Fifty thousand documents a day, uneven file sizes and bursts from batch uploads.

Decisions

Use asynchronous processing, isolate untrusted files, version outputs and route uncertain or high-risk results to review.

Failure to examine

A parser or model failure causes repeated processing, duplicate records or the loss of the original evidence.

Trade-off

A faster extraction path may require more human correction; stronger validation increases processing and review cost.

What You Produce

For the capstone, bring these into one production AI system design. Explain how the design changes when traffic rises, a provider fails, an access rule changes or a cost assumption proves wrong. A prototype or measurement can support a decision; the main submission is the architecture and its justification.

• ·A problem statement, functional requirements and measurable quality attributes.
• ·System-context, component, data-flow and sequence diagrams.
• ·Capacity estimates, an inference strategy and an operating-cost model.
• ·ADRs comparing the chosen design with credible alternatives.
• ·A threat model, failure analysis and recovery plan.
• ·An evaluation and observability plan tied to user and system outcomes.

An Example of an Architecture Decision Record

Illustrative decision: Use incremental, versioned indexing for frequently updated policy documents, with a scheduled reconciliation job.

Reason: A full nightly rebuild would not meet the assumed freshness target.

Cost of the decision: More work is needed for duplicate events, deletion handling, index consistency and monitoring.

Evidence to collect: Update lag, deletion propagation, authorization tests, reconciliation failures and processing cost. Revisit the decision if the workload or freshness requirement changes.