How Mux Evolved Stream.new Content Moderation from Primitive Vision APIs to Advanced Multimodal AI

The evolution of automated content moderation on the open web reflects a broader industry challenge: balancing the open accessibility of user-generated platforms with the operational necessity of filtering out policy-violating material. For Mux, the video infrastructure company behind the anonymous, free-to-use video-sharing demo application stream.new, this challenge has prompted three distinct generational leaps in content moderation infrastructure since the application’s launch in 2020. As platforms offering zero-friction, account-free uploading inevitably attract a spectrum of unintended content—ranging from pirated motion pictures and professional sports broadcasts to explicit material—engineering teams are forced to move beyond rudimentary frame-sampling toward context-aware, multimodal artificial intelligence systems.

The journey of stream.new highlights how automated moderation architectures have transformed over a five-year period. Driven by the introduction of proprietary toolkits and native API integrations, Mux has successfully streamlined its compliance operations while shielding human review teams from the internet’s most problematic media.

The Genesis of Stream.New and the 2021 Moderation Baseline

Launched in 2020, stream.new was designed as the simplest possible expression of Mux Video. The platform allowed any user to record or upload a video directly in the browser and instantly receive a shareable link, completely free of charge and without requiring user registration. Over subsequent years, the application processed hundreds of thousands of files, serving simultaneously as a reference implementation for developers, a public-facing dogfooding platform for new Mux features, and a magnet for unpredictable user traffic.

Because the service imposed no authentication barriers, it naturally attracted a diverse array of edge-case and prohibited uploads. Alongside legitimate user tests, the platform routinely ingested adult content, pirated feature films, reaction-style watchalong videos, and an anomalously high volume of professional cycling footage.

To manage this influx, the engineering team deployed a traditional moderation pipeline in 2021. Following standard industry practices of the era, the system extracted sample thumbnails from across uploaded video assets and transmitted them to external vision APIs. Initially, Mux relied on a dual-provider approach, utilizing Google Vision SafeSearch and Hive. This redundancy was implemented out of an abundance of caution; because automated content moderation for user-generated video was still relatively unchartered territory for the team, maintaining two distinct providers mitigated the risk of false positives and false negatives, ensuring that a user’s legitimate video was rarely deleted without cross-verification.

However, this foundational architecture presented notable operational friction. Maintaining two bespoke API clients required reconciling disparate response formats—such as Hive’s granular float scores versus Google’s five-point likelihood enumerations—resulting in roughly 250 lines of custom code and associated test suites. Furthermore, the system orchestration executed synchronously within the platform’s asset.ready webhook handler. Lacking an asynchronous task queue or dedicated workflow orchestration layer, slow responses from third-party vision providers frequently triggered timeouts and necessitated error retries.

More critically, static thumbnail classification possessed a fundamental limitation: it could identify what an individual video frame looked like, but it could not comprehend what the video actually was. A frame extracted from a pirated television broadcast often appeared entirely innocuous to a SafeSearch algorithm. Consequently, while the automated tier successfully intercepted overt policy violations, a massive volume of complex infractions still fell upon human moderators monitoring a dedicated Slack channel.

Technical Debt and the 2024–2025 Modernization Initiative

Before any sophisticated AI plumbing could be upgraded, the underlying application infrastructure required a comprehensive overhaul. By late 2024, stream.new operated on an aging technology stack comprising Node 16, React 17, and Next.js 12 utilizing the legacy Pages Router. Modern AI tooling and workflow orchestration packages proved incompatible with this environment.

In early 2025, a series of targeted pull requests modernized the application stack, elevating the environment to Node 20, React 18, and Next.js 13 with the App Router, subsequently paving the way for migrations to Next.js 16 and Node 22. While infrastructural modernization is frequently overlooked in high-level product narratives, engineering leads emphasize that maintaining legacy environments past their support lifecycle introduces compounding technical debt that impedes security and feature velocity. Much of this migration work was heavily assisted by large language models, allowing a single developer to review and safely merge sweeping dependency upgrades with minimal friction.

Migration Phase One: The Introduction of @mux/ai

In December 2025, Mux released @mux/ai, an open-source TypeScript toolkit designed to bridge the gap between video assets and multimodal AI providers. The package standardized thumbnail extraction intervals, transcript fetching and cleaning, structured prompting, and type-safe evaluation results. By February 2025, stream.new integrated @mux/ai, fundamentally altering its backend architecture.

The migration allowed engineers to purge both bespoke provider clients and the complex score-reconciliation logic that had been maintained for years. In their place, the application deployed streamlined asynchronous calls running inside durable Vercel Workflow instances. While the dual-provider strategy was maintained—incorporating OpenAI’s moderation API alongside Hive—the underlying plumbing was entirely abstracted by the new toolkit.

The most transformative capability unlocked during this phase was question-based multimodal analysis, executed via the askQuestions feature. While binary classification models excel at detecting explicit or violent imagery, stream.new’s most persistent moderation hurdles involved contextual policy violations—such as unauthorized full-length films or synchronized reaction videos—that appeared benign to traditional frame scrapers.

Modern multimodal models, provided with sufficient temporal context and speech-to-text transcripts, are capable of answering complex analytical prompts regarding video content. Consequently, Mux translated its institutional moderation policies into a standardized array of plain-English queries evaluated against every upload:

• "Is this a professionally produced full length movie or TV show, or a standalone segment from it?"
• "Is this professionally produced footage of a cycling race?"
• "Is this a watchalong-style video where a person or small group is actively watching and reacting to a full-length movie or TV episode as the main focus of the clip?"
• "Does this video use offensive language, and/or is likely to offend?"
• "Does this contain explicit slurs, dehumanization, or threats toward a protected group?"
• "Is this video mostly of feet?"

These queries mapped directly to recurring enforcement challenges. For example, watchalong videos—where a creator records themselves in a corner of the frame while copyrighted anime plays in the background—previously bypassed NSFW filters because the dominant visual element of the sampled frame featured a seated human. Multimodal analysis, however, synthesized broader contextual clues to identify the underlying violation.

The integration of automated language detection for auto-generated captions further supercharged this capability. Previously, developers were required to explicitly declare the spoken language of an incoming video to generate accurate transcripts—an impossible prerequisite for anonymous uploads. By implementing automatic language detection across all assets, the moderation workflow paused execution until caption tracks were compiled, supplying the AI analysis engine with a comprehensive transcript regardless of the source language. To maintain operational clarity within English-speaking review channels, developers subsequently contributed an outputLanguageCode parameter to @mux/ai, ensuring summaries and reasoning outputs were consistently rendered in English.

Structured responses containing binary determinations, confidence ratings, and model justifications enabled automated enforcement. When confidence scores for defined prohibited categories exceeded established thresholds, the system automatically purged the asset and logged the rationale to internal communication channels, reserving human review for ambiguous edge cases.

Migration Phase Two: Transitioning to Native Mux Robots

While @mux/ai provided a powerful open-source solution, it operated as a bring-your-own-key toolkit requiring developers to provision their own API credentials and manage orchestration within their individual infrastructure. The natural progression of this technology was to operationalize the engine natively within the core Mux video platform.

In April 2025, stream.new completed its second major AI migration by adopting Mux Robots—a first-party video intelligence API that executes analysis natively alongside the primary video asset. Rather than managing third-party API keys within application code, stream.new initiated moderation tasks via native Mux SDK endpoints:

const  id  = await mux.robotsPreview.jobs.moderate.create(
  parameters: 
    asset_id: assetId,
    thresholds:  sexual: 0.85, violence: 0.85 ,
    sampling_interval: 10,
    max_samples: 25,
  ,
);

To handle asynchronous job processing efficiently within serverless environments without resorting to inefficient polling loops, engineers leveraged Vercel Workflow’s hook primitives. By establishing a unique cryptographic hook keyed to the specific asset ID prior to initiating the moderation job, the workflow suspends execution until the completion webhook arrives. Upon receipt, the webhook payload—including all analytical outputs—is delivered directly back into the workflow stream.

This architectural shift allowed Mux to entirely remove external provider API keys from the application codebase. Mux Robots dynamically allocated optimal vision and language models behind the scenes, abstracting provider management away from the application layer entirely.

Production Tuning and Strategic Implications

Fine-tuning the platform’s moderation parameters involved striking a delicate operational balance. On a free, public service where users bear no financial stake, false positives—erroneously removing a legitimate user upload—carry a negative user experience cost. However, allowing policy-violating or copyrighted material to remain active introduces significant legal and reputational risks.

Through continuous observation of internal review channels, engineering teams calibrated their auto-deletion thresholds to a confidence level of 0.85, paired with a frame-sampling interval of 10 seconds. This calibration provided the optimal intersection of high interception accuracy and minimal false-positive rates for unauthenticated uploads.

The evolution of stream.new’s moderation architecture underscores a broader industrial trend: the consolidation of auxiliary artificial intelligence services directly into core cloud infrastructure providers. By shifting from cobbled-together multi-vendor API clients to open-source abstraction layers, and finally to native platform automation primitives like Mux Robots, modern development teams can drastically reduce codebase complexity while simultaneously expanding analytical capabilities.

As Mux continues to roll out advanced orchestration frameworks such as Mux Robots Directives, applications operating at the edges of the open web will increasingly rely on autonomous, self-contained AI pipelines to maintain digital safety standards with minimal human overhead.