Zillow Unveils Scalable Notification Engine Processing 100 Million Daily Messages Using AWS Step Functions
Real estate technology leader Zillow has successfully engineered and deployed a high-throughput, configuration-driven communication orchestration system capable of processing more than 100 million personalized email and push notifications daily. Developed in close collaboration with Amazon Web Services (AWS), the new infrastructure radically transforms how the company engages home shoppers by replacing fragmented, legacy microservice architectures with a unified serverless pipeline powered by AWS Step Functions Express Workflows and the Universal Content Generator (UCG).
The architectural overhaul resolves long-standing inefficiencies that previously encumbered Zillow’s product engineering teams. Historically, the company relied on more than 10 disparate microservices to drive over 20 distinct notification campaigns, ranging from saved search alerts and property updates to seller reports and custom home recommendations. Each legacy generator operated its own isolated workflow, data models, business rules, and assembly logic. As Zillow expanded its product ecosystem, this decentralized approach generated substantial engineering duplication, inconsistent user experiences, inflated operational overhead, and rigid release cycles that hindered rapid experimentation.
Architectural Evolution and Strategic Objectives
The mandate for centralization stemmed from a shifting operational trade-off within Zillow’s Comms Experience division. While initial system designs prioritized isolated scaling challenges, the proliferation of distinct products revealed that identical features were frequently reimplemented and retested across multiple campaign services. This fragmentation created high barriers to rolling out platform-wide capabilities and complicated maintenance workflows, monitoring dashboards, log aggregations, and on-call support rotations.
To overcome these roadblocks, Zillow conceptualized a centralized model governed by two primary pillars: the Campaign Orchestrator and the Universal Content Generator. The Campaign Orchestrator functions as a configuration-driven orchestration layer executing short-lived workflows on Step Functions Express Workflows. Simultaneously, the Universal Content Generator serves as a centralized library of reusable, schema-driven content modules that handle data hydration, hidden home filtration, property detail retrieval, propensity scoring, and A/B testing logic.
Competitive Evaluation and Technology Selection
Before committing to the AWS serverless ecosystem, Zillow’s engineering leadership conducted a comprehensive competitive evaluation of alternative workflow engines, including Temporal, Flowable, custom in-house engines, and both Standard and Express modes of AWS Step Functions. The evaluation measured platforms against four core criteria: operational reliability, total cost of ownership, development velocity, and multi-team scalability.
Open-source and self-hosted options such as Temporal and Flowable were ultimately excluded because they would have introduced an entirely new infrastructure footprint requiring
dedicated operational management, server fleets, and worker pools. This ran counter to Zillow’s primary strategic goal of reducing release and monitoring overhead. Similarly, building an in-house workflow engine offered maximum architectural control but demanded disproportionate custom lifecycle management, worker queue integrations, and monitoring overhead—estimated at roughly four times the build and maintenance effort required by AWS Step Functions.
Step Functions emerged as the definitive choice for the minimum viable product (MVP), delivering required orchestration capabilities at approximately a quarter of the anticipated build effort and ongoing maintenance cost. Furthermore, because Step Functions is entirely serverless, product engineering teams were able to initiate pilot projects instantaneously without provisioning underlying compute infrastructure.
Balancing Volume, Latency, and Cost Through Express Workflows
Zillow’s production workloads operate at extreme velocity. The initial deployment model scaled rapidly from an initial baseline of 14 million daily workflows to more than 100 million execution requests at peak volume, translating to thousands of individual transactions per second. Because real-time notifications require sub-second delivery to remain relevant to active home shoppers, execution latency and throughput capacity were paramount.
Comparing the operational characteristics of Step Functions Standard versus Express modes, Zillow determined that Express Workflows were uniquely suited for the high-throughput, short-lived nature of notification assembly. While Standard Workflows are optimized for durable, long-running, exactly-once orchestrated tasks lasting hours or days, Express Workflows accommodate high-frequency, short-duration tasks—completing well within the five-minute execution limit—and utilize at-least-once execution semantics paired with idempotent module calls.
To maximize cost efficiency at scale, the engineering team introduced an advanced batching mechanism within the routing layer. Rather than initiating a distinct workflow execution for every individual message, the system groups multiple notification payloads into a single batched Step Functions execution. This optimization substantially reduces total API request volume, minimizes StartExecution rate consumption, and dramatically lowers AWS billing overhead. Because batch execution runtimes scale sub-linearly relative to payload size, processing a batch of ten messages consumes roughly the same memory footprint while achieving up to a 90 percent cost reduction compared to ten individual workflow runs.
Rigorous Validation and Continuous Integration Testing
Implementing a dynamic, configuration-driven workflow architecture necessitated robust automated testing safeguards. Because a significant portion of campaign behavior is defined through declarative JSONata expressions and modular configurations, Zillow instituted strict continuous integration (CI) validation protocols.
Instead of relying on offline string evaluations, the development team utilizes the AWS Step Functions TestState API to execute unit tests against the identical runtime environment utilized in production. Shared JSONata mapping files and schema definitions are continuously validated against expected module inputs and outputs. Any malformed payload, runtime expression error, or configuration anomaly immediately halts deployment pipelines, mitigating downstream production risks while empowering non-technical stakeholders to safely modify campaign configurations.
Organizational Impact and Future Outlook
The deployment of the Campaign Orchestrator and UCG architecture has yielded profound organizational and technical benefits for Zillow. Product development cycles for new messaging campaigns and customer communication features have been dramatically accelerated. Rather than writing dedicated microservices or modifying multiple legacy codebases, product teams now configure campaigns declaratively using validated schemas and shared modular components.
Operational maintenance demands have also plummeted. Ongoing system maintenance overhead decreased from approximately eight hours per week to just two hours, while initial development estimates were compressed from 32 weeks down to nine weeks. Moreover, strict logical isolation guarantees that individual campaign queues operate independently via dedicated worker pools and circuit breakers, ensuring that transient traffic spikes or downstream outages within a single campaign cannot propagate across the broader messaging infrastructure.
Industry analysts note that Zillow’s implementation of a configuration-driven, serverless orchestration layer establishes a benchmark architectural pattern for modern enterprise applications processing high-volume, short-lived event streams. By leveraging native cloud primitives such as AWS Step Functions Express Workflows, EventBridge connections, and direct HTTP invocation models, organizations can achieve massive horizontal scalability without incurring the traditional operational burdens associated with self-managed distributed systems.
