MLOps Projects · AWS implementation

AWS MLOps Project with Amazon SageMaker

Follow the model lifecycle from source and data identity through SageMaker deployment, operational evidence and rollback boundaries.

Written byAshutosh· AI InstructorVerified byVivek· AIOps and Generative AI InstructorUpdated2026-08-19Partially tested

How do you build an MLOps project on AWS?

An AWS MLOps project connects versioned code and data to repeatable training, evaluation, model registration, controlled deployment, telemetry and rollback. Amazon SageMaker can own several managed stages, while S3, IAM, CI/CD and monitoring still require explicit contracts. This page separates the parts verified by School of Core AI from current AWS documentation and planned validation.

Difficulty: advanced
Environment: AWS
Last verified: 2026-08-19

Project objective and acceptance criteria

Source code is versioned in Git with tagged releases
Dataset is stored in S3 with versioning enabled
Training runs on SageMaker with logged parameters and metrics
Model is registered in the SageMaker Model Registry with approval status
Endpoint is deployed and reaches InService state
Prediction test verifies the endpoint returns correct outputs
IAM roles follow least-privilege principles
Cleanup procedure is documented and tested

AWS MLOps architecture

SageMaker owns training, registration, and deployment. S3, IAM, and CloudWatch are supporting AWS services that require explicit configuration.

Source, dataset, container and artifact identity

Field	Value	Evidence
Source commit	Git SHA tagged at pipeline trigger	Tested by SCAI
Dataset version	S3 object version ID for training data	Officially documented
Container image	SageMaker training image with pinned dependencies	Partially tested
Training run	SageMaker training job name with hyperparameters	Partially tested
Model version	SageMaker Model Registry version with approval status	Officially documented
Endpoint	SageMaker endpoint name with configuration	Partially tested
Monitoring baseline	CloudWatch metrics for latency and invocations	Officially documented
Rollback target	Previous model version + endpoint configuration	Planned validation

Training and experiment evidence

SageMaker QuickSetup Domain creationTested by SCAI

QuickSetup Domain created successfully. Studio accessible. Account details redacted.

Date: 2026-08-19
Environment: AWS SageMaker Studio (account redacted)
Result: Domain operational. Notebook execution confirmed.

SageMaker Serverless Inference endpoint creation at 1024 MBTested by SCAI

Endpoint reached InService state. Scikit-learn model served successfully. Test predictions returned correct outputs.

Date: 2026-08-19
Environment: AWS SageMaker Serverless Inference (account redacted)
Result: Endpoint operational at 1024 MB memory configuration.

SageMaker Serverless Inference endpoint at 4096 MBTested by SCAI

Timeouts and ResourceLimitExceeded errors occurred. Endpoint did not reach InService at 4096 MB.

Date: 2026-08-19
Environment: AWS SageMaker Serverless Inference (account redacted)
Result: Failed at 4096 MB. Documented as a limitation.

Limitation: The 4096 MB configuration exceeded available serverless quota in the test account.

Evaluation and model registration

After training, the model is evaluated against a held-out test set. Metrics are logged and the model is registered in the SageMaker Model Registry with a version number and approval status. The registry provides staging, production, and archived states for managing the model lifecycle.

Registration is the gate between training and deployment. An unapproved model cannot be deployed. The approval workflow can be manual or automated via CI/CD.

Serverless endpoint verification

SCAI tested a scikit-learn model on a SageMaker Serverless Inference endpoint at 1024 MB memory. The endpoint reached InService state and served test predictions correctly. This is partially-tested evidence — the serverless endpoint works for a simple model, but real-time endpoints with GPU instances were not tested due to account limitations.

Serverless inference is suitable for intermittent traffic and small models. For high-throughput or GPU-dependent models, real-time endpoints with dedicated instances are required. Access to ml.m5.large instances was denied in the SCAI test account, limiting real-time endpoint testing.

Failures encountered and what they prove

Serverless endpoint timeout at 4096 MB memory

Tested by SCAI

Signal:: ResourceLimitExceeded and timeout errors during endpoint creation
Root cause:: The 4096 MB memory configuration exceeded available serverless quota in the test account
Containment:: Reduced memory to 1024 MB, which reached InService successfully
Durable fix:: Document account quota limits before selecting serverless configuration. Request quota increase if 4096 MB is required.

ml.m5.large instance access denied

Tested by SCAI

Signal:: Access denied error when attempting to create a real-time endpoint with ml.m5.large
Root cause:: The SCAI test account does not have permission for ml.m5.large instances
Containment:: Real-time endpoint testing was paused. Serverless endpoint was used instead.
Durable fix:: Request instance access via AWS support or use a different account with broader instance permissions.

KServe on EKS validation not completed

Tested by SCAI

Signal:: EKS cluster setup required additional permissions not available in test account
Root cause:: Account limitations prevented EKS cluster creation for KServe testing
Containment:: KServe validation was paused and is not part of the current evidence set
Durable fix:: Use an account with EKS permissions or a dedicated Kubernetes testing environment.

CloudWatch telemetry and operational checks

CloudWatch provides endpoint telemetry including invocation count, latency, error rate, and model latency. These metrics are available without additional configuration for SageMaker endpoints.

Important: Amazon SageMaker Model Monitor closed to new customers effective 2026-07-30. Existing customers may continue using it, but AWS does not plan new features. For new customers, endpoint telemetry plus an open-source drift detection job is the current alternative. Do not present Model Monitor as a universally available recommendation.

CloudWatch metrics: Invocations, ModelLatency, OverheadLatency, Invocation4XXErrors, Invocation5XXErrors
Model Monitor: closed to new customers as of 2026-07-30 — verify before recommending
Alternative: endpoint telemetry + custom drift detection job using SageMaker Processing

IAM, secrets and network exposure

IAM roles should follow least-privilege principles. The SageMaker execution role should only access the S3 buckets needed for training data and model artifacts. Network exposure should be restricted using VPC endpoints and security groups where possible.

Secrets — API keys, database credentials — should never be stored in environment variables or code. Use AWS Secrets Manager or Parameter Store for secret management.

Quotas, cost controls and cleanup

Resource envelope: SageMaker Serverless Inference at 1024 MB, SageMaker Studio domain, S3 storage for datasets and artifacts. No GPU instances were used due to account limitations.

Numbers are estimated as of 2026-08-19.

Quotas encountered

Serverless Inference memory: 1024 MB succeeded, 4096 MB failed with ResourceLimitExceeded
ml.m5.large instance access denied — real-time endpoint testing not possible
EKS cluster creation not available in test account — KServe validation paused

Cleanup requirements

Delete SageMaker endpoints after testing to avoid ongoing charges
Delete SageMaker Studio domain when not in use
Clean up S3 model artifacts and training outputs
Remove IAM roles and policies created for the project

Monitoring options for new AWS customers

Since Model Monitor is closed to new customers, new AWS accounts need alternative monitoring approaches. The current recommendation is to use CloudWatch metrics for infrastructure telemetry and a custom drift detection job using SageMaker Processing or a scheduled Lambda function.

This is not an AWS-endorsed pattern — it is a practical alternative based on current availability. Re-check the official Model Monitor availability page before implementing, as AWS policies may change.

Repository and artifact map

Path	Responsibility	Availability
sagemaker-training/	SageMaker training job scripts and configuration	private
sagemaker-deployment/	Serverless endpoint deployment scripts	private
sagemaker-registry/	Model registration and approval workflow	private
evidence/	Run logs, endpoint screenshots, and test outputs (redacted)	private
iam/	IAM role definitions and policy documents	private

Sources and last verification

Amazon SageMaker AI MLOps
Amazon Web Services · Accessed 2026-08-19
Supports: SageMaker provides managed MLOps capabilities.
https://aws.amazon.com/sagemaker/ai/mlops/
Last verified: 2026-08-19 · official-documentation
SageMaker Model Registry Version Management
Amazon Web Services · Accessed 2026-08-19
Supports: Model registry provides version management and approval workflows.
https://docs.aws.amazon.com/sagemaker/latest/dg/model-registry-version.html
Last verified: 2026-08-19 · official-documentation
SageMaker Serverless Inference Endpoints
Amazon Web Services · Accessed 2026-08-19
Supports: Serverless inference endpoints scale to zero.
https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html
Last verified: 2026-08-19 · official-documentation
Model Monitor Availability Change
Amazon Web Services · 2026-07-30
Supports: Model Monitor closed to new customers 2026-07-30.
https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-availability-change.html
Last verified: 2026-08-19 · official-documentation
SageMaker QuickSetup Domain Execution
School of Core AI · 2026-08-19
Supports: SCAI used SageMaker QuickSetup Domain for lab execution.
Last verified: 2026-08-19 · scai-execution
Serverless Endpoint at 1024 MB
School of Core AI · 2026-08-19
Supports: Serverless endpoint reached InService at 1024 MB.
Last verified: 2026-08-19 · scai-execution
Serverless Timeout at 4096 MB
School of Core AI · 2026-08-19
Supports: Timeouts occurred at 4096 MB configuration.
Last verified: 2026-08-19 · scai-execution
ml.m5.large Access Denied
School of Core AI · 2026-08-19
Supports: Access to ml.m5.large was denied in test account.
Last verified: 2026-08-19 · scai-execution

Execute the SageMaker capstone with guided support

The course includes the real AWS SageMaker capstone scope supported by current account access, with instructor review and guided lab sequences.

Execute the SageMaker lifecycle with current account access
Debug endpoint deployment failures with instructor guidance
Produce a reviewed SageMaker deployment artifact

Explore the MLOps course