We introduce partially-stateful data-flow, a new streaming data-flow model that supports eviction and reconstruction of data-flow state on demand. By avoiding state
explosion and supporting live changes to the data-flow
graph, this model makes data-flow viable for building
long-lived, low-latency applications, such as web applications. Our implementation, Noria, simplifies the backend infrastructure for read-heavy web applications while
improving their performance.
A Noria application supplies a relational schema and a
set of parameterized queries, which Noria compiles into
a data-flow program that pre-computes results for reads
and incrementally applies writes. Noria makes it easy
to write high-performance applications without manual
performance tuning or complex-to-maintain caching layers. Partial statefulness helps Noria limit its in-memory
state without prior data-flow systems’ restriction to windowed state, and helps Noria adapt its data-flow to
schema and query changes while on-line. Unlike prior
data-flow systems, Noria also shares state and computation across related queries, eliminating duplicate work.
On a real web application’s queries, our prototype
scales to 5× higher load than a hand-optimized MySQL
baseline. Noria also outperforms a typical MySQL/memcached stack and the materialized views of a commercial
database. It scales to tens of millions of reads and millions of writes per second over multiple servers, outperforming a state-of-the-art streaming data-flow system