Adding Automatic Subtitles to Video Pipelines With AWS MediaConvert

Adding subtitles to a video used to require a separate transcription and editing workflow. A video would be uploaded, an audio track would be transcribed, subtitle timings would be created, and the final subtitle file would then be attached to the video.

That workflow still works, but it creates additional processing steps.

AWS MediaConvert can be used with speech transcription workflows to generate subtitles as part of a managed video processing pipeline. This is useful when an application needs to process many uploaded videos and automatically produce subtitle tracks without manually editing every file.

The practical architecture usually combines Amazon S3, AWS Elemental MediaConvert, and a speech-to-text service such as Amazon Transcribe.

This article explains how to design that workflow, where MediaConvert fits, how subtitle files are handled, and what developers should consider before putting automated subtitle generation into production.

The Basic Architecture

A typical automated subtitle workflow looks like this:

Video Upload
     |
     v
Amazon S3
     |
     v
MediaConvert / Video Processing
     |
     v
Speech Transcription
     |
     v
Subtitle File
     |
     v
MediaConvert
     |
     v
Output Video + Subtitle Track
     |
     v
Amazon S3

The exact orchestration can vary, but the responsibilities should remain clear.

The video processing service handles media conversion and packaging, while the transcription service produces the text and timing information required for subtitles.

Why Automate Subtitle Generation?

Manual subtitle creation becomes expensive when an application handles hundreds or thousands of videos.

Consider a training platform where users upload:

10 videos per day
50 videos per day
500 videos per day

A manual workflow quickly becomes a bottleneck.

Automation allows the application to process videos asynchronously.

User uploads video
        |
        v
Upload completes
        |
        v
Processing job created
        |
        v
Video processed
        |
        v
Speech transcribed
        |
        v
Subtitle generated
        |
        v
Final asset published

The user does not need to remain connected while the processing happens.

Store the Original Video in Amazon S3

The first step is usually uploading the source video to Amazon S3.

A simple object structure might look like:

s3://video-platform/
    uploads/
        course-1001/
            lesson-01.mp4

Keep the original object separate from generated files.

For example:

s3://video-platform/
    uploads/
    processed/
    subtitles/

This separation makes lifecycle management and troubleshooting easier.

It also prevents an application from accidentally overwriting the original source video.

Detecting a New Video

An application can trigger processing after a new video arrives in S3.

A common event-driven architecture is:

S3 Upload
    |
    v
S3 Event
    |
    v
AWS Lambda / EventBridge
    |
    v
Processing Workflow

The trigger should create a processing job rather than trying to perform the entire video workflow synchronously.

For example:

Upload completed
        |
        v
Create job
        |
        v
Return response to user

The user interface can then display:

Uploaded
Processing
Transcribing
Generating subtitles
Ready

This is much better than keeping an HTTP request open while a video is being processed.

Where AWS MediaConvert Fits

MediaConvert is a managed video processing service.

It can be used to:

Convert video formats
Create different resolutions
Process audio tracks
Generate output files
Package video content
Include subtitle tracks in output media

The important architectural point is that MediaConvert does not need to perform every step itself.

A subtitle pipeline may look like:

Source Video
     |
     v
Extract / Process Audio
     |
     v
Speech Transcription
     |
     v
Subtitle File
     |
     v
MediaConvert
     |
     v
Final Video

Keeping these responsibilities separate makes the pipeline easier to maintain.

Generate the Transcript

The speech recognition stage converts the audio into text with timing information.

For example, the resulting transcript may conceptually contain:

00:00:01.200 --> 00:00:04.000
Welcome to the course.

00:00:04.100 --> 00:00:07.500
In this lesson, we will build an API.

The exact transcript format depends on the transcription service and workflow.

The important information is:

Text
Start time
End time
Speaker information when available
Confidence information when supported

Subtitle generation depends heavily on the timing data.

A transcript without useful timing information cannot produce a properly synchronized subtitle track.

Subtitle Formats

Different video workflows use different subtitle formats.

Common formats include:

WebVTT
SRT
TTML
Embedded subtitle tracks

The format should be selected based on how the video will be consumed.

For example, a browser-based video player may use WebVTT.

A downloadable subtitle file may use SRT.

An enterprise media workflow may require another format depending on the packaging and playback environment.

Do not choose the subtitle format only because it is easy to generate. Choose it based on the final playback requirements.

Example WebVTT Output

A simplified WebVTT file looks like this:

WEBVTT

00:00:01.200 --> 00:00:04.000
Welcome to the course.

00:00:04.100 --> 00:00:07.500
In this lesson, we will build an API.

The timestamps determine when the text appears and disappears.

A subtitle generation service should therefore preserve timing information carefully.

Even small timing errors can make subtitles difficult to follow.

Configure the MediaConvert Job

Once the video and subtitle file are available, create a MediaConvert job that references the source media and subtitle input.

A typical workflow is:

Input Video
    |
    +-- Video Settings
    +-- Audio Settings
    +-- Subtitle Input
             |
             v
        MediaConvert
             |
             v
      Output Video
             +
      Subtitle Track

The exact MediaConvert job configuration depends on the required output format, codec, packaging type, and subtitle delivery method.

For example, a web application may require an MP4 output plus a separate subtitle file, while another workflow may package the subtitles with the video stream.

Creating the Job From Code

A backend service can create MediaConvert jobs using the AWS SDK.

A simplified C# structure looks like this:

using Amazon.MediaConvert;
using Amazon.MediaConvert.Model;

var client = new AmazonMediaConvertClient();

var request = new CreateJobRequest
{
    Role = mediaConvertRoleArn,
    Settings = new JobSettings
    {
        // Configure input, output,
        // audio, video, and subtitles here.
    }
};

var response = await client.CreateJobAsync(request);

The actual JobSettings configuration depends on the media workflow.

For production applications, avoid putting a large MediaConvert configuration directly inside application code.

A configuration object or template is usually easier to maintain.

Use Job Templates

A video platform may generate thousands of jobs with similar settings.

Instead of rebuilding every setting for every request, define reusable job templates.

For example:

Course Video Template
    |
    +-- H.264 output
    +-- Standard resolution
    +-- WebVTT subtitles

Mobile Video Template
    |
    +-- Mobile resolution
    +-- Optimized bitrate
    +-- Subtitle track

The application then supplies only the values that change between jobs.

This reduces configuration errors and makes changes easier to manage.

Tracking Processing Status

Video processing is asynchronous.

The application should maintain a processing record such as:

Video ID
Source Location
MediaConvert Job ID
Transcript Status
Subtitle Status
Output Location
Processing Status
Error Message
Created At
Completed At

For example:

video-1001
    |
    +-- Uploaded
    +-- Transcription started
    +-- Transcript ready
    +-- Subtitle generated
    +-- MediaConvert started
    +-- Completed

This gives developers enough information to troubleshoot a failed pipeline.

Use an Orchestration Layer

As the workflow becomes more complex, directly chaining Lambda functions can become difficult to maintain.

For example:

Upload
  |
  v
Transcription
  |
  v
Subtitle conversion
  |
  v
MediaConvert
  |
  v
Publishing

An orchestration service such as AWS Step Functions can represent these stages explicitly.

A simplified state flow looks like:

Start
  |
  v
Check Input
  |
  v
Start Transcription
  |
  v
Wait for Transcript
  |
  v
Create Subtitle
  |
  v
Start MediaConvert
  |
  v
Wait for Completion
  |
  v
Publish Video
  |
  v
Complete

This makes retries and failure handling easier to manage.

Handling Transcription Delays

Transcription does not necessarily complete immediately.

The application should not assume:

Start transcription
     |
     v
Transcript immediately available

Instead, treat transcription as an asynchronous operation.

For example:

Transcription Started
        |
        v
Status Check
        |
        +---- In Progress
        |
        +---- Failed
        |
        +---- Completed

If the transcription service supports event notifications, use them instead of repeatedly polling whenever practical.

If polling is required, use controlled intervals and stop after a defined timeout.

Synchronizing Subtitle Timing

Subtitle quality depends on more than transcription accuracy.

The timing must also match the spoken content.

Consider:

Audio:
"Welcome to the course."

Subtitle:
"Welcome to the course."

If the subtitle appears two seconds too early, the viewer experience is still poor even though the text is correct.

Therefore, test:

First subtitle timing
Long sentences
Rapid speech
Multiple speakers
Pauses
Background noise
Very short phrases
Long videos

Timing quality should be part of the test plan.

Handling Multiple Languages

A video platform may eventually need subtitles in multiple languages.

The pipeline can be extended:

Source Video
     |
     v
Transcript
     |
     +---- English
     |
     +---- Hindi
     |
     +---- Spanish
     |
     +---- French

The translation stage should be treated separately from transcription.

Do not assume that translating raw transcript text automatically produces high-quality subtitles.

The workflow should preserve timing information while translating the text.

For example:

Original timing
      |
      v
English transcript
      |
      v
Translated text
      |
      v
Localized subtitle file

This allows multiple subtitle tracks to be generated from the same source video.

Handling Speaker Identification

Some videos contain multiple speakers.

Examples include:

Interviews
Panel discussions
Training sessions
Podcasts
Meetings

If the transcription workflow provides speaker identification, the subtitle pipeline can use that information where appropriate.

However, displaying speaker names on every subtitle may make the screen difficult to read.

A practical approach is to define a presentation rule.

For example:

Speaker 1:
Welcome to the session.

Speaker 2:
Thanks for joining.

Whether speaker labels should appear depends on the content and playback experience.

Large Video Files

Large video files should not be passed through application servers unnecessarily.

Prefer:

Client
   |
   v
S3 Upload
   |
   v
Processing Services

rather than:

Client
   |
   v
Application Server
   |
   v
Application Server uploads video to S3

Direct S3 upload reduces application server load.

For large files, multipart upload can also improve reliability.

The application can issue a presigned upload URL and allow the client to upload directly to the appropriate S3 location.

Security Considerations

Video files may contain private training material, customer recordings, or internal meetings.

Protect the pipeline accordingly.

Use:

Private S3 buckets
Least-privilege IAM roles
Encryption
Controlled access to output files
Short-lived access URLs where appropriate
Logging and monitoring

Avoid granting the MediaConvert role access to unrelated S3 buckets.

The processing role should have only the permissions required for the media workflow.

Handling Failed Jobs

Every production media pipeline needs explicit failure handling.

Possible failures include:

Invalid video format
Missing audio track
Transcription failure
Invalid subtitle format
MediaConvert job failure
Permission errors
Missing S3 objects
Unsupported media settings

Do not simply mark the video as failed.

Record enough information to identify the failing stage.

For example:

Video ID: video-1001

Status: Failed
Stage: MediaConvert
Job ID: 123456
Reason: Invalid subtitle configuration
Retryable: Yes

This makes operational support much easier.

Retry Strategy

Not every error should be retried.

For example:

Temporary service error
    -> Retry

Network timeout
    -> Retry

Invalid video file
    -> Do not retry automatically

Missing permissions
    -> Fix configuration first

Invalid subtitle data
    -> Regenerate subtitle

Use bounded retries with backoff for transient failures.

Otherwise, a broken job can create repeated processing attempts and unnecessary costs.

Cost Considerations

Video processing costs depend on the amount and type of media processed.

Monitor:

Number of videos
Video duration
Output formats
Number of subtitle tracks
Transcription usage
MediaConvert processing
S3 storage
Data transfer

For example, generating five different video resolutions for every upload may increase processing costs even if most users watch only one resolution.

Start with the required outputs and add additional processing only when there is a real product need.

Common Mistakes

Processing Video Synchronously

Do not keep an HTTP request open while transcription and video processing occur.

Use asynchronous jobs.

Sending Large Videos Through the Application Server

Upload directly to S3 where possible.

Treating Transcription as Instantaneous

Transcription is an asynchronous process.

Track its status explicitly.

Hardcoding MediaConvert Settings

Job templates are easier to manage when multiple videos use the same output configuration.

Ignoring Subtitle Timing

Correct text with incorrect timing still produces poor subtitles.

Giving MediaConvert Excessive Permissions

Use a dedicated IAM role with only the permissions required by the workflow.

Retrying Permanent Failures

Invalid input does not become valid because the same job is executed again.

Classify errors before retrying.

Production Checklist

Before deploying an automated subtitle pipeline, verify:

Videos are uploaded directly to protected S3 storage.
Upload events trigger asynchronous processing.
Transcription status is tracked.
Subtitle format matches the target player.
Subtitle timing is tested with real videos.
MediaConvert jobs use reusable configurations where appropriate.
Processing status is persisted.
Failed jobs record the failing stage and error.
Retry rules distinguish transient and permanent failures.
IAM roles use least privilege.
Output files are protected from unauthorized access.
Large video processing is tested.
Cost metrics are monitored.
Multiple-language workflows preserve subtitle timing.
Completed outputs can be replayed or regenerated when necessary.

Advantages and Disadvantages

Advantages

Automates subtitle generation for uploaded videos.
Removes manual subtitle creation from the normal workflow.
Scales better than manual processing.
Works well with S3-based video pipelines.
MediaConvert handles managed video processing.
Subtitle tracks can be integrated into broader media workflows.

Disadvantages

The pipeline requires multiple AWS services.
Transcription quality depends on audio quality and language.
Subtitle timing requires testing.
Large video processing can increase costs.
Failed asynchronous jobs require monitoring and retry logic.
Multi-language subtitles introduce additional processing stages.

When This Architecture Makes Sense

This approach is useful for applications such as:

Online learning platforms
Corporate training systems
Video publishing platforms
Webinar platforms
Internal knowledge systems
User-generated video applications
Video archives

It is especially useful when the application receives videos continuously and needs subtitles without manual intervention.

For a small number of videos, manual processing may still be simpler. The value of automation increases as video volume and the number of required output formats grow.

Summary

Automated subtitle generation is best implemented as an asynchronous media pipeline rather than a single application request.

Amazon S3 can provide durable source and output storage, a speech-to-text service can generate the transcript and timing information, and AWS Elemental MediaConvert can handle the video processing and subtitle integration required by the final output.

The most important production concerns are not just generating text. The pipeline must also handle timing, failures, retries, security, large files, processing status, and cost.

Start with one video format and one subtitle language. Once the workflow is reliable, extend it to multiple resolutions, languages, and output formats as the application requires.