Joe McCraw.

Case study

Caption Theater

Turning unused letterbox space around cinema-aspect-ratio video into a premium, unobstructed reading area for captions

Active · open source
View on GitHub

The problem

Many ultra-widescreen films and shows are presented on 16:9 screens with black letterbox space above and below the active picture. That space just sits there, unused, while viewers are forced to choose between watching the scene and racing to read dense, fast-moving captions overlaid on top of it.

The solution

I designed and built Caption Theater as a modular playback add-on that repurposes that empty letterbox space into a dedicated, unobstructed reading area. When conditions are safe, the active picture shifts into a theater-style, top-pinned layout, and captions render in the open lower region — with recently-displayed cues persisting briefly so viewers have more time to read without losing the scene.

Caption Theater opt-in prompt shown over a video, offering 'Use Caption Theater' or 'Standard playback' buttons
The feature is opt-in: viewers are prompted before anything changes, and can decline in favor of standard playback.
Caption Theater active: the video is pinned to the top of the screen (green border) with an unobstructed caption reading band below (blue border) showing clean, multi-line dialogue text
Caption Theater mode: the picture stays fully visible and undistorted while captions get their own dedicated, readable space.

Design principles

Three product values drive the architecture:

  • Readable dwell time — cues that have already appeared can remain visible briefly, giving viewers more time to scan them.
  • Context preservation — the active picture stays visible and undistorted while recent caption context stays available in a stable reading region.
  • Trustworthy fallback — if the system can’t prove the experience is safe, it returns to native playback and native caption behavior, no exceptions.

The project treats subtitle and caption text as meaningful content rather than decoration, and separates viewport eligibility (is this content shaped right for the layout?) from subtitle-format eligibility (can we safely parse and time these cues?). It never reveals future dialogue, never stretches or crops the picture, and never touches ad or legal overlays.

What I built

  • Active-picture boundary detection using pixel analysis and provider/QC metadata for content without direct frame access
  • A runtime safety and fallback system that returns to native playback whenever visual safety, subtitle semantics, ad state, or platform support is uncertain
  • Multi-speaker caption presentation with retained-cue history
  • Comprehensive debug tooling that explains every activation and fallback decision
  • Native tvOS and macOS targets built on AVFoundation and AVKit-adjacent integrations, processing HLS metadata and timed WebVTT captions

Status

Active open-source project, currently shipping native tvOS (the primary living-room prototype) and macOS (a demo/QA surface for offline HLS playback and layout review) targets, with iOS as a roadmap target. Licensed under Apache License 2.0.