Skip to content

Independent architecture study

Uploaded video ready for viewers

Automated Video-on-Demand Pipeline on AWS

A media library needs uploaded videos converted, published and retained without manually coordinating every job. This independent architecture study follows AWS guidance for an orchestrated video-on-demand pipeline, with distribution formats for different devices and lifecycle archival for source files.

An independent architecture study based on public AWS guidance. A design exercise, not a client engagement.

Workload
File-based video ingestion, transcoding and publishing
Constraint
Device compatibility, visible processing failures and growing source storage
Approach
Step Functions coordinates media conversion, publishing, notifications and archival

What makes this difficult

  • A source is not a stream

    An uploaded master file needs suitable distribution formats before viewers on different devices can play it through the delivery network.

  • Jobs cross service boundaries

    Ingesting, converting and publishing a video involves separate services, so progress and errors need a coordinated workflow rather than manual handoffs.

  • Originals keep accumulating

    Keeping every large source file in immediately accessible storage can grow expensive even when published renditions handle most viewing activity.

Follow an uploaded video

A source file moves through coordinated conversion and publishing while the workflow records progress and retains outputs.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  1. 01

    Start from an uploaded master

    Receives sourcevideoAmazon S3CoordinatesproductionStep FunctionsRuns steps anderrorsAWS Lambda

    A source video arrives in S3 and enters the ingest workflow, where Step Functions coordinates Lambda steps so processing follows an explicit production sequence.

    Illustrative example Illustrative example: an uploaded training video enters the production workflow.

  2. 02

    Make files viewers can play

    Runs steps anderrorsAWS LambdaCreates HLS andDASHMediaConvertStores playbackfilesAmazon S3

    A Lambda step starts MediaConvert, which transcodes the source into distribution formats such as HLS and DASH and writes the resulting playback files to S3.

    Illustrative example Illustrative example: the source video produces playback files for different devices.

  3. 03

    Keep progress and errors visible

    Runs steps anderrorsAWS LambdaCreates HLS andDASHMediaConvertRecords jobmetadataAmazon DynamoDBLogs encodingeventsAmazon CloudWatchPublishes jobnoticesAmazon SNS

    Workflow functions record metadata and publish notifications while MediaConvert emits encoding events to CloudWatch, giving operators evidence to inspect when processing succeeds or fails.

    Illustrative example Illustrative example: a job record and encoding notice describe the processing outcome.

  4. 04

    Choose the delivery path

    Stores playbackfilesAmazon S3Packages whenselectedMediaPackageDelivers toviewersAmazon CloudFront

    CloudFront delivers stored playback files to viewers, with MediaPackage available as an optional packaging path when the distribution requirements call for packaging at delivery time.

    Illustrative example Illustrative example: the published video is delivered from stored renditions or the optional packaging path.

  5. 05

    Retain the workflow output

    CoordinatesproductionStep FunctionsQueues workflowoutputsAmazon SQS

    Step Functions sends workflow outputs to SQS, keeping a queued handoff for downstream handling separate from the media files delivered to viewers through CloudFront.

    Illustrative example Illustrative example: a completed workflow leaves an output message for downstream handling.

  6. 06

    Move older masters into archive

    Receives sourcevideoAmazon S3Archives sourcemastersS3 Glacier

    The S3 lifecycle policy transitions source media to Glacier over time, reducing the need to retain every original in the same storage class as active distribution files.

    Illustrative example Illustrative example: an older source master transitions to archival storage.

  • Request
  • Data
  • Later, queued
  • Control

The hard part

The original and the published video have different lives

The source master and distribution files serve different purposes. MediaConvert creates playback renditions for viewers, while the original remains subject to its own S3 lifecycle policy. Archiving that source does not move the distribution files into archive. This separation supports a growing library, although future reprocessing must account for retrieval of an archived master.

Keeps the sourcemasterAmazon S3Creates playbackfilesMediaConvertStores publishedcopiesAmazon S3Archives theoriginalS3 GlacierServes publishedcopiesAmazon CloudFront
Illustrative: published playback files remain available while the source master later moves to archive.

Why this design

  1. Use managed media conversion

    Chosen
    Use MediaConvert to produce HLS and DASH distribution formats.
    Because
    The business needs multi-device playback without operating its own encoding fleet.
    Cost
    Output formats, resolution, codecs and features influence transcoding charges and configuration complexity.
    Reconsider when
    Assess self-managed encoding when specialised processing requirements justify the operational burden.

  2. Model production as coordinated steps

    Chosen
    Use Step Functions to coordinate Lambda work across ingest, processing and publishing.
    Because
    Explicit stages make job progress and error handling easier to reason about across services.
    Cost
    The workflow introduces state-machine design and coordination between separate functions.
    Reconsider when
    Simplify the workflow when the media task is limited enough to be handled safely by a single managed job.

  3. Archive source masters separately

    Chosen
    Apply an S3 lifecycle policy that transitions source media to Glacier.
    Because
    Published playback files can serve viewers while original masters follow a different retention lifecycle.
    Cost
    Reprocessing an archived source requires planning for the selected archive retrieval behaviour and charges.
    Reconsider when
    Keep sources immediately accessible when frequent editing or re-encoding makes retrieval delays unsuitable.

Cost and operating assumptions

  • Transcoding outputs

    MediaConvert charges use output minutes with normalisation factors for resolution, codecs and features, under the applicable service tier.

  • Optional packaging

    MediaPackage adds charges when the optional packaging path is selected for distribution.

  • Source and rendition storage

    S3 stores sources and playback files, while Glacier archival introduces storage and retrieval choices for retained masters.

  • Delivery and coordination

    CloudFront transfer to viewers contributes delivery cost, alongside usage of Step Functions, Lambda, DynamoDB, CloudWatch, SNS and SQS.

Assumptions behind the numbers

No monthly total is published for this guidance; pricing depends on usage, region and options, including output profiles, viewing traffic, retention and archive retrieval.

How it stays reliable

  • Handle errors within the workflow

    Step Functions coordinates the production stages, with Lambda functions performing step work and processing error messages.

  • Record job evidence

    DynamoDB stores workflow metadata and CloudWatch records logs and MediaConvert notifications for investigation.

  • Notify and hand off outcomes

    SNS publishes encoding, publishing and error notifications, while SQS captures workflow outputs for downstream handling.

  • Plan for archived-source recovery

    Design consideration: validate the retrieval process for archived masters before relying on them for future reprocessing or restoration.

When this is a poor fit. A poor fit for live broadcasting or a small video library whose requirements are already met by a managed publishing platform.

Full technical detailThe numbered AWS flow, every service and the Well-Architected notes.

How the AWS guidance describes it

  1. Amazon S3 and Amazon S3 Glacier. Source media is stored in S3 and eventually transitions to Glacier according to an S3 lifecycle policy.
  2. AWS Step Functions. State machines orchestrate ingest, processing and publishing workflows.
  3. AWS Elemental MediaConvert. MediaConvert transcodes source media into distribution formats such as HLS and DASH.
  4. AWS Lambda. A collection of functions performs workflow step work and processes error messages.
  5. Amazon DynamoDB. DynamoDB stores metadata captured throughout the workflow.
  6. Amazon CloudWatch. CloudWatch provides logging, with CloudWatch Events rules handling MediaConvert notifications.
  7. Amazon SNS. SNS topics publish encoding, publishing and error notifications.
  8. AWS Elemental MediaPackage. Optional MediaPackage processing creates streams formatted for several devices from a single video input.
  9. Amazon S3. A distribution bucket stores the processed media files.
  10. Amazon CloudFront. A CloudFront distribution delivers video content to end users.
  11. Amazon SQS. An SQS queue captures workflow outputs.

Well-Architected

Operational Excellence
Design reasoning: orchestration, metadata, logs and notifications make processing stages and failures available for inspection.
Security
Design reasoning: access controls must cover source and distribution buckets as well as the CloudFront delivery path.
Reliability
Design reasoning grounded in the flow: Lambda error handling and queued workflow outputs support explicit handling of processing outcomes.
Performance Efficiency
Design reasoning grounded in the guidance: managed transcoding creates suitable renditions and CloudFront delivers them from the edge.
Cost Optimisation
Design reasoning grounded in the flow: source lifecycle archival separates long-term retention from active playback storage.
Sustainability
Design reasoning: event-driven processing aligns compute with submitted videos, while lifecycle policies manage long-term source storage.