UReal-time Data @ Uber

SIGMOD '21 — ACM International Conference on Management of Data (2021)arXiv:2104.00087 ↗

Real-time Data
Infrastructure at Uber

Kafka, Flink, Pinot, HDFS and Presto — re-engineered for trillions of messages a day

Yupeng Fu and Chinmay Soman (Uber Technologies)

This is the paper that explains how Uber turns trillions of Kafka messages and petabytes of data per day into decisions made in seconds — surge prices, fraud checks, restaurant dashboards, ML monitoring. Rather than building proprietary systems, Uber assembled an entire real-time stack from open source — Apache Kafka for streaming storage, Flink for stream processing, Pinot for sub-second OLAP, HDFS for archival, Presto for interactive queries — then re-engineered each layer to survive three kinds of scaling: data, use cases, and users. It reads as a field guide to the freshness-vs-consistency trade-offs, multi-region failover machinery, and self-service tooling behind one of the industry's largest real-time deployments.

Read the PDFuReplicator on GitHub

PBs

of data per day

collected across all regions

trillions

of messages / day

one of the largest Kafka fleets

99.99%

availability SLA

with seconds-level freshness

< 1 s

p99 query latency

Pinot, at 10,000s of QPS

Introduction

The paper opens by framing real-time analytics as mission-critical rather than nice-to-have: customer incentives, fraud detection, and ML model prediction all require decisions in seconds over streams that reached trillions of messages and petabytes per day (October 2020). The real-time ecosystem decomposes into three broad areas: a messaging platform ( transport), , and for near-real-time analytical queries.

Three scaling axes structure everything that follows: scaling data (exponential year-over-year volume growth from several thousands of microservices), scaling use cases (new business verticals with competing requirements — a financial dashboard and a log pipeline want opposite trade-offs), and scaling users (a technical-skill spectrum from operations staff with no engineering background to advanced platform engineers). Each open-source adoption — Kafka, Flink, Pinot, HDFS, Presto — is judged against all three.

challenge 1

Scaling data

Incoming real-time volume grows exponentially year over year, produced by several thousands of microservices — trillions of messages, petabytes daily.

challenge 2

Scaling use cases

New business verticals keep arriving with competing requirements — a financial ledger and a log pipeline want opposite trade-offs from the same platform.

challenge 3

Scaling users

The people querying this data range from operations staff with no engineering background to platform engineers — the tooling must serve the whole spectrum.

Requirements

Seven requirement categories, each with a concrete anchor: consistency (mission-critical financial dashboards need zero data loss inter- and intra-region), availability (99.99% guarantee — an outage directly impacts the business), data freshness (most use cases need events queryable seconds after production), query latency (user-facing tools need p99 under 1 second), scalability (petabytes/day with continuous organic growth), cost (Uber is a low-margin business), and flexibility (both programmatic APIs and SQL-style declarative interfaces, push- and pull-based).

The section closes with the structural insight of the paper: by , consistency and freshness/availability cannot both be guaranteed at Uber's scale. Dynamic pricing explicitly prioritizes freshness over consistency — late-arriving data is simply excluded from time-window computations — while financial use cases invert the choice. The whole architecture is a menu of such per-use-case prioritizations over the same five systems.

Surge pricingFreshness, availability (active-active)Consistency — late messages dropped from windows; lossy high-throughput KafkaKafka → Flink (ML) → key-value store
Restaurant ManagerQuery latency, freshnessQuery flexibility — aggressive Flink pre-aggregation fixes the query shapesKafka → Flink (roll-ups) → Pinot
Prediction monitoringScalability (100Ks of time series, millions of points/sec)Query-time flexibility — pre-aggregated OLAP cubesKafka → Flink (join + aggregate) → Pinot
Eats ops automationReliability, exploration→production continuityLatency — minutes-level aggregation windows sufficePinot ← Presto (ad-hoc → productionized rules)
Financial dashboardsConsistency — zero data loss (active-passive)Freshness — offset-synced failover over instant availabilityAggregate Kafka → single named consumer
How each use case prioritizes the §2 requirements — the paper's central argument that one stack can serve competing needs through per-pipeline configuration (synthesized from §2, §5 and §6).

Abstractions & System Overview

Section 3 defines seven logical layers with minimum requirements. Storage: generic blob/object store, read-after-write consistency, optimized for high write rates (reads are for bootstrap and ). Stream: interface with partitioning and delivery. Compute: arbitrary computation over stream + storage, at-least-once semantics. OLAP: limited SQL over fresh data (filters, aggregations, group/order by), with ingestion via primary keys for critical uses. SQL: full SQL over OLAP and compute (joins included). API: programmatic access for advanced users. Metadata: versioned, backward-compatible schema management for all layers.

Figure 3 of the paper maps abstractions to systems: Kafka (stream + intermediate storage), Flink (compute), Pinot (OLAP), HDFS (archival storage), Presto (SQL). The interesting property is what sits between layers: schema services, ingestion frameworks, and SQL compilers that let the same logical query or dataset move across systems — the glue that the later sections describe.

3.1 · The seven layers

Seven abstraction layers — click any layer

Stream → Apache Kafka

Publish-subscribe over partitioned topics: producers append, any subscriber consumes event-by-event. Minimum requirements: data partitioning and at-least-once delivery.

Redrawn from Figure 2: layers stacked bottom-up (Storage → API), with Metadata spanning them all. Each layer states minimum guarantees; each open-source system occupies one (SQL spans two paths).

3.2 · The layers, made concrete

The real-time stack — an event's journey

Producersrider/driver appsmicroservicesDB changelogsApache Kafkastreaming storageregional + aggregateclustersApache Flinkstream processing1000s of jobs,most via FlinkSQLApache Pinotreal-time OLAPHDFSarchival · Avro→ParquetPrestointeractive SQLConsumersdashboards · apps · rulesevent → queryable in seconds · trillions of messages/day · every block is open source

1 · Produce

Several thousands of microservices — plus rider/driver apps and database changelogs — publish trillions of messages and petabytes of data per day into Kafka. Topics auto-provision when services deploy; federation hides the physical clusters (each kept under ~150 nodes).

Redrawn from the paper's Figure 3 (system overview). Click any component or press play to trace an event from production to query.

Apache Kafka

Stream + intermediate storage

Pub-sub backbone: app events, DB changelogs, lake ingestion.

Uber's additions: Cluster federation, dead letter queues, Consumer Proxy (gRPC push), uReplicator + Chaperone.

Apache Flink

Compute

Stateful stream processing with checkpointing.

Uber's additions: FlinkSQL (Calcite), resource estimation + auto-scaling, rule-based auto-healing, 3-layer job platform.

Apache Pinot

OLAP

Sub-second analytical serving on fresh data.

Uber's additions: Upsert (shared-nothing), peer-to-peer segment recovery, schema inference, FlinkSQL sink.

HDFS

Storage (archival)

Source of truth: Avro landing → Parquet for engines.

Uber's additions: Also holds Flink checkpoints, Pinot segment archives, and all backfill history.

Presto

SQL (interactive)

In-memory MPP engine for ad-hoc full SQL.

Uber's additions: Pinot connector with projection / aggregation / limit pushdown (planner + Connector API extensions).

3.3 · The stack in numbers

Kafka throughput

trillions / day

Messages per day across all regions (Oct 2020) — one of the largest Kafka deployments in the industry.

Data volume

PBs / day

Raw analytical data continuously collected from riders, drivers and eaters across all regions.

Producers

1000s of microservices

Several thousands of microservices produce into the stream layer; topics auto-provision on deploy.

Availability SLA

99.99%

Platform guarantee — unavailability directly impacts business functions like driver payments.

Data freshness

seconds

Events must be queryable seconds after production for most use cases.

Query latency

< 1 s

User-facing analytical queries (e.g., Restaurant Manager) target p99 under one second.

Kafka cluster size

< 150

Empirically ideal broker count per physical cluster — federation stitches clusters into one logical cluster.

Flink workload

1000s of jobs

Flink is the de facto stream processor at Uber, growing ~30% year over year.

OLAP footprint

100s of TBs

Grown from dozens of GBs in two years; query load from hundreds to tens of thousands of QPS.

ML monitoring

100Ks of time series

Millions of data points per second — beyond Uber's dedicated time-series database.

Kafka retention

a few days

Why backfill must run from HDFS/Hive instead of replaying Kafka (the Kappa problem).

Client languages

4

Java, Python, Go, NodeJS — why thin generated clients and the Consumer Proxy exist.

Apache Kafka for Streaming Storage

Kafka carries trillions of messages and PBs/day, feeding both real-time consumers and the Hadoop lake (event data from rider/driver apps, database , microservice logs). Four Uber additions: — a metadata server exposes a logical cluster, routing producers/consumers to physical clusters (empirically ideal at < 150 nodes), enabling transparent topic migration and horizontal growth without client restarts. — after repeated consumer failures, messages publish to a DLQ topic for later purge or on-demand retry, so a poison message can't stall trip-receipt processing.

— with tens of thousands of consuming applications, thick client libraries became unmanageable; the proxy consumes on behalf of users and pushes via gRPC to registered endpoints, machine-generated thin clients replace the consumer library, and the push model both improves throughput and escapes Kafka's partition-count ceiling on consumer-group parallelism. — Uber's open-sourced cross-cluster replicator with a rebalancing algorithm that minimizes partition movement and adaptively sheds bursts to standby workers; audits every pipeline stage by comparing windowed message counts and alerting on mismatch (zero-data-loss evidence).

Consumer Proxy — Kafka's pull becomes gRPC push

Kafka topicpartition 0partition 1partition 2partition 3Consumer Proxyfull Kafka client · retries · acksconsumer service (10,000s of apps)thin gRPC clientsDLQ topicpurge or retry on demandconsumegRPC pushnack → retryafter N retries

1 · Proxy consumes, so apps don't have to

The Consumer Proxy subscribes to the topic's partitions using the full-featured Kafka client — the last place that complexity lives. Applications keep only a thin, machine-generated gRPC client (Java, Python, Go or NodeJS).

Redrawn from Figure 4. With tens of thousands of consuming applications in four languages, the sophisticated Kafka client lives in one place — the proxy — and everyone else gets push delivery, retries and a DLQ for free.

Apache Pinot for Real-time OLAP

Pinot was adopted in 2018 on benchmark evidence: vs Elasticsearch, 4× lower memory, 8× lower disk, 2–4× faster queries (filter/aggregation/group-order-by mixes); vs Druid (architecturally similar), Pinot's bit-compressed forward indices and specialized indices (star-tree, sorted, range) gave order-of-magnitude latency wins on some workloads. Growth in 2 years: dozens of GBs → several hundreds of TBs; hundreds of QPS → tens of thousands of QPS.

Uber's four contributions: Upsert — the only open-source real-time OLAP store with it; input streams are partitioned by primary key so all records for a key land on one node, giving a shared-nothing design (no centralized coordination) with better scalability, no single point of failure, easier operation. Full SQL via the Presto integration (§4.5) — Pinot freshness + Presto joins/subqueries. Ecosystem integration — schema inference from Kafka samples (with cardinality estimation), a FlinkSQL sink, and Piper/Spark workflows building offline Pinot tables from Hive. Peer-to-peer segment recovery — replacing synchronous archival through a single controller (a scalability bottleneck and freshness violation) with async archival + recovery from server replicas.

Elasticsearch's resource cost, as a multiple of Pinot

Same data ingested into both systems (Uber benchmark, §4.3). Dashed line = Pinot's footprint (1×).

Pinot (baseline, 1×) Elasticsearch overhead

With identical data, Elasticsearch used 4× the memory and 8× the disk, and answered filter/aggregation/group-by queries 2–4× slower. The bar shows the top of the reported range for latency.

Two years of growth

Data footprint

dozens of GBsseveral hundreds of TBs

≈4 orders of magnitude in 2 years

Query workload

a few hundred QPStens of thousands of QPS

≈2 orders of magnitude in 2 years

Recovery, redesigned

Segment recovery, before / after — drag to compare

Uber replaced Pinot's centralized segment store with a peer-to-peer scheme: same consistency, better freshness, no single-controller bottleneck.

Redesign — async + peer-to-peerserver 1segment replicasserver 2segment replicasserver 3segment replicasfailed replica? recover from peersdeep store (HDFS/S3)archival is async —off the hot path
Original — controller in the write pathserver 1segment replicasserver 2segment replicasserver 3segment replicascontroller ×1deep store (HDFS/S3)sync upload —bottleneck +freshness risk
⇄
ORIGINAL PINOTUBER'S REDESIGN

HDFS Archival & Presto Interactive Queries

HDFS (§4.4) is the archival source of truth: Kafka streams land as Avro, compaction jobs rewrite them to Parquet for Hive/Presto/Spark. HDFS also backs (new pipelines, ML training over months of history, bug-fix reprocessing), Flink checkpoints (stream offsets + per-container operator state snapshots), and Pinot segment archival (replica repair, server bootstrap).

Presto (§4.5) provides interactive full SQL via an in-memory engine that avoids intermediate materialization. Uber exploited Presto's Connector API with its high-performance I/O interface to integrate Pinot as a backend: initially only predicate , then — after extending Presto's query planner and Connector API — projection, aggregation and limit pushdown into Pinot. The combination yields sub-second PrestoSQL latencies over seconds-fresh data, 'not possible on standard backends such as HDFS/Hive,' and supports the exploration→production workflow of §5.4.

Use Cases Analysis

Surge pricing (§5.1): Kafka → Flink (complex ML algorithm over trip events, rider/driver status in time windows, per-hexagon geofences) → key-value store for fast multiplier lookup. Prioritizes freshness + availability over consistency: late messages are excluded from windows, Kafka runs in high-throughput (not lossless) configuration, and the pipeline runs active-active across regions (§6). Restaurant Manager (§5.2): owner-facing dashboards (satisfaction, popular items, sales, service quality) with strict latency needs and fixed query patterns — Flink preprocessors do 'aggressive filtering, partial aggregates and roll-ups' so Pinot meets the latency SLA, trading query flexibility for speed (transformation-time vs query-time processing).

Prediction monitoring (§5.3): joins model predictions to observed outcomes for live accuracy tracking; with thousands of models × hundreds of features, that's several hundreds of thousands of time series and millions of data points per second — beyond Uber's internal time-series database — handled by Flink's horizontal scaling plus pre-aggregated OLAP cubes in Pinot. Eats ops automation (§5.4): COVID-era capacity compliance; Presto-on-Pinot ad-hoc queries validated insights, then the same queries were productionized in a rule engine aggregating location statistics over minutes and alerting couriers/restaurants — 'a seamless path from ad-hoc exploration to production rollout.' Table 1 maps which abstraction layers each use case exercises.

APIclient libraries✓—✓—
SQLFlinkSQL / PrestoSQL—✓✓✓
OLAPApache Pinot—✓✓✓
ComputeApache Flink✓✓✓✓
StreamApache Kafka✓✓✓✓
StorageHDFS—✓✓—
Table 1 (faithful to the paper) — the abstraction-layer components exercised by each of the four example use cases. Compute and Stream are load-bearing for all four; every layer serves at least two.

How many of the four use cases touch each layer

Derived from Table 1 — Compute (Flink) and Stream (Kafka) carry every single use case

Every abstraction layer serves at least two of the four flagship use cases — the paper's evidence that the whole stack is load-bearing, with Kafka and Flink as the universal substrate.

The life of one event

  1. t = 0

    A rider taps, a courier accepts, a driver moves

    The event is produced into a regional Kafka cluster — one of thousands of microservices publishing trillions of messages a day. The topic was auto-provisioned when the service deployed.

  2. milliseconds

    Replicated everywhere

    uReplicator copies the event into aggregate Kafka clusters in every region, giving each region the global view. Chaperone counts messages at every hop — proof that nothing was lost.

  3. seconds

    Flink computes

    Streaming jobs window, join and aggregate: the surge model updates a hexagon's multiplier; the Restaurant Manager pre-aggregations roll up orders; prediction monitoring joins the event to an ML forecast.

  4. seconds later

    Queryable in Pinot

    Results land in Pinot (upserted by primary key) or a key-value store — the event is now part of every dashboard aggregate, seconds after it happened.

  5. sub-second

    Someone — or something — acts

    A restaurant owner's dashboard refreshes (p99 < 1 s), a Presto-powered ops rule fires an alert, a surge multiplier changes. The loop from physical world to decision closed in seconds.

All-Active Strategy

Multi-region Kafka: regional clusters replicate into aggregate clusters in each region (via uReplicator), giving every region a global view. Active-active (surge): each region runs an identical, computationally intensive Flink job over its aggregate feed; an all-active coordinating service designates a primary region whose 'update service' writes multipliers to an active-active key-value store. On regional failure another region is promoted — the design recomputes state everywhere because the state is too large to replicate synchronously across regions; convergence follows from identical aggregate inputs.

Active-passive (consistency-critical, e.g., payments/auditing): a single uniquely-identified consumer reads from the aggregate cluster in the primary region only. Failover hinges on cross-region offset mapping: uReplicator 'periodically checkpoints the offset mapping from source cluster to destination cluster in an active-active database,' and an offset sync job continuously synchronizes consumer offsets between regions. On failover, the standby's consumer resumes from the latest synced offset — no data loss and no excessive reprocessing backlog.

Surviving a region failure — two strategies

all-active coordinatorRegion A — PRIMARYregional Kafkaaggregate Kafkaglobal viewFlink jobsurge multipliersupdate serviceRegion B — standby (computing too)regional Kafkaaggregate Kafkaglobal viewFlink jobsame computationupdate serviceactive-active DBmultiplier lookupuReplicator · both directionsprimary writesBoth regions recompute the same state from the same aggregate feed — state is too big to replicate, so it converges instead.
Redrawn from Figure 6: an all-active coordinating service designates a primary; every region runs the identical, computationally intensive Flink job on the aggregated feed, so failover is just promotion.

Backfill

Backfill triggers: testing new pipelines against historical data, ML training over months of history, and reprocessing after bugs. Lambda (parallel batch + streaming stacks) duplicates logic and invites divergence; Kappa (replay from Kafka alone) demands retention far beyond Uber's few-day Kafka retention — infeasible at PB/day. Uber's two Flink-based solutions: SQL-based — the same FlinkSQL query compiles into two jobs, a streaming job on Kafka (DataStream API) and a batch job on Hive (DataSet API); Lambda-shaped, but with a single user-maintained implementation.

Kappa+ (API-based): reuse the streaming job's code as-is, reading directly from archived Hive/HDFS data with small configuration changes. The hard parts the paper enumerates: identifying boundaries of bounded input, throttling the effectively infinite read rate of historical data so job state and downstream systems survive, and handling out-of-order data — archived data isn't ordered like the original stream, requiring larger watermark/buffering windows. Both approaches are declared 'an active area of investigation with lots of edge cases.'

One query → two Flink jobs

The same FlinkSQL query compiles into a streaming job (DataStream API, reading Kafka) and a batch job (DataSet API, reading Hive). Lambda-shaped execution, but the user maintains exactly one implementation — the compiler owns the divergence risk instead of the engineer.

Best for: the FlinkSQL crowd — anyone whose pipeline is already SQL.

Related Work & Lessons Learned

Related work (§8) positions the stack against: traditional messaging (ActiveMQ, RabbitMQ, Oracle EMS, IBM MQ — outperformed by Kafka's ecosystem; Pulsar noted as promising but less mature), stream processors (Storm, Samza, Heron, Spark Streaming, Apex; proprietary Photon, Facebook Puma/Swift, Kinesis), OLAP (Druid, ClickHouse), SQL systems (Hive, Dremel, Spark SQL, Impala, Drill; Procella, Trill), and full proprietary stacks (Facebook's realtime infra, Google F1 Lightning, Alibaba Hologres) — Uber's differentiator being commitment to open source and community leverage.

Lessons (§9) in four groups. Open-source adoption: each system was built for a narrow purpose; Uber re-worked them across use cases, languages (RESTful Kafka clients for Java/Python/Go/NodeJS), infrastructure, and security. Rapid evolution: interface standardization, monorepo review, thin clients (Kafka client upgrades used to take months), language consolidation (Java/Go for infra, PrestoSQL for analytics), and CI/CD continuously staging open-source updates. Operations: declarative automation of cluster management, config generation, containerization, plus per-use-case dashboards, alerts and chargeback. Onboarding/debugging: central metadata + lineage for data discovery; event auditing (unique IDs, timestamps, service name, tier stamped on events, tracked across Kafka→Flink→Pinot→Hive in every datacenter to spot loss/duplication); auto-provisioned, quota-enforced topics; drag-and-drop Flink/Pinot pipeline deployment.

lesson 1

Open source is a head start, not a solution

Kafka in 2014 was a Java log-shipper. Uber added REST clients in 4 languages, throttling, metadata discovery, DLQs and federation before it fit. Budget for the gap between 'adopted' and 'works here.'

lesson 2

Thin clients, or upgrades take quarters

Fleet-wide Kafka client upgrades used to take months. Thin, machine-generated clients + interface standardization + a monorepo turned library migrations from projects into deploys.

lesson 3

Automate operations before they outgrow the team

Machines multiply faster than engineers: declarative cluster management, generated configs, containerization — plus per-use-case dashboards, alerts and chargeback so owners see their own cost and health.

lesson 4

Self-service or die by support ticket

Topics auto-provision on deploy; schemas are inferred and centrally discoverable with lineage; events carry ID/timestamp/service/tier so loss is detectable everywhere; pipelines deploy by drag-and-drop.

Conclusion & Future Work

Conclusion: the open-source-based stack processes multiple petabytes/day with years of reliable production operation. Against the three challenges — data: thin Kafka clients, federation, Flink checkpointing, Pinot peer-to-peer recovery let thousands of microservices ride one platform (among the industry's largest Kafka deployments); use cases: one consumer protocol spans logging (availability-first) to financial data (zero-loss); Pinot serves both low-latency OLAP and Presto-powered exploration; users: FlinkSQL puts data scientists' and ops personnel's pipelines in production within hours, PrestoSQL federates queries, and backfill needs no custom code.

Future work, four items: unified stream/batch processing (users currently express the same logic twice, in different languages on different frameworks); multi-region and multi-zone deployments with the challenge to 'optimize data placement to balance redundancy for reliability against the storage cost of excessive copies'; on-prem/cloud agnosticism so the stack moves freely as Uber's cloud posture evolves; and tiered storage for Kafka and Pinot (cost efficiency via cold tiers, elasticity via storage-serving separation), pursued with the open-source communities.

The 2021 wishlist, scored from the future

Unified stream + batch

Users write the same logic twice, in two languages, on two frameworks.

Realized: Flink's unified runtime, the lakehouse wave.

Multi-region & multi-zone

Balance data redundancy for reliability against the storage cost of excessive copies.

Still the open cost/reliability frontier.

Cloud agnosticism

Abstract away DC/cloud specifics so the stack moves freely.

Realized: table stakes for every data platform.

Tiered storage

Cold data on cheap media; separate storage from serving for elasticity.

Realized: shipped in both Kafka (KIP-405) and Pinot.

Code & Resources

uber/uReplicator→

The open-sourced cross-cluster Kafka replicator from §4.1 — Helix-based controller, minimal-movement rebalancing, adaptive burst handling. The offset-mapping checkpoints it writes are what make the active-passive failover of §6 lossless.

apache/pinot→

The OLAP store of §4.3 — Uber's upsert design and the peer-to-peer segment recovery described in this paper both live upstream now, alongside the star-tree, sorted and range indices behind the benchmark numbers.

The whole stack, as annotated pseudocode

The paper describes an architecture rather than an algorithm; this pseudocode traces one event through every layer — Kafka, Flink(SQL), Pinot, Presto, and backfill — preserving the reported numbers and design choices from §3–§7.

# Uber's real-time stack, end to end — annotated pseudocode (paper sections 3-7).
# The paper describes architecture, not code; this traces one event's life
# through every layer, preserving the reported numbers.

def rider_opens_app(event):                    # ~trillions of events/day
    """Producers: thousands of microservices, 4 client languages."""
    kafka.produce(topic="trip-events", value=event, acks="throughput")
    # Surge config favors throughput over lossless-ness (freshness > consistency).
    # Regional cluster (<150 nodes; federation adds clusters behind one
    # logical cluster) -> uReplicator -> aggregate cluster in EVERY region.
    # Chaperone audits per-stage windowed counts: zero-loss evidence.

def surge_pricing_job(stream):                 # Apache Flink, active-active
    """Same job runs in each region on the same aggregate feed."""
    for window in stream.window(minutes=2, late_data="drop"):
        multipliers = ml_pricing_model(window.group_by(hexagon))
        kv_store.put(multipliers)              # served if this region is primary

# Meanwhile, an analyst ships a dashboard pipeline in HOURS, in SQL only:
FLINK_SQL = """
    SELECT restaurant_id, COUNT(*) AS orders, window_start
    FROM eats_orders                            -- Kafka topic (schema inferred)
    GROUP BY restaurant_id, TUMBLE(rowtime, INTERVAL '5' MINUTE)
"""
# Calcite: SQL -> logical plan -> optimized physical plan -> Flink job.
# Platform does the ops the analyst can't: stateless jobs are CPU-bound,
# stream-stream joins memory-bound -> auto-scaling + rule-based healing.
flink.submit(FLINK_SQL, sink=pinot.table("orders", upsert_key="order_id"))
# Pinot upsert: partition stream by primary key -> same node, shared-nothing.

def restaurant_dashboard(owner):               # p99 < 1 s over seconds-fresh data
    return presto.query(                       # planner pushes work INTO Pinot:
        "SELECT ... FROM pinot.orders WHERE ...",
        pushdown=["predicate", "projection", "aggregation", "limit"])

def backfill(job, months=6):                   # Kafka keeps only a few days
    """Kappa+: unchanged streaming code over the HDFS/Hive archive."""
    return job.run(source=hive("trip-events"), bounded=True,
                   throttle=True, watermark_slack="large")  # out-of-order history

Glossary

Every dotted-underlined term on this page links back here. Hover any of them in context, or browse the full definitions below.

Apache Kafka

A distributed, partitioned, replicated commit log. Producers append messages to topics; consumers read them independently at their own pace. At Uber it is the transport for app events, database changelogs and lake ingestion — one of the largest deployments in the industry.

Apache Pinot

A columnar OLAP datastore that ingests directly from Kafka and serves filter/aggregation/group-by queries at tens of thousands of QPS. Uber contributed upsert support and peer-to-peer segment recovery upstream.

Presto

A massively parallel processing (MPP) query engine performing all computation in memory. Uber extended its planner and Connector API to push projection, aggregation and limit down into Pinot, achieving sub-second full SQL over fresh data.

HDFS

Stores all raw Kafka data (Avro, compacted to Parquet), Flink checkpoints, and Pinot segment archives. Every backfill ultimately reads from here, because Kafka retains only a few days of data.

OLAP

The workload class of dashboards and monitoring: scan-and-aggregate queries over recent data, as opposed to point lookups (OLTP). The paper defines OLAP as its fourth abstraction layer, implemented by Pinot.

Stream processing

Applying computational logic (aggregation, joins, ML inference) to unbounded event streams with results in seconds. The paper's Compute abstraction layer, implemented by Flink.

Publish-subscribe

The decoupling pattern of the Stream layer: producers don't know their consumers. The paper's minimum requirements for this layer are data partitioning and at-least-once delivery.

At-least-once

The baseline delivery guarantee of the Stream and Compute layers: no loss, but retries can duplicate. Systems needing correctness on top deduplicate downstream — e.g., Pinot's exactly-once ingestion via primary keys.

Exactly-once ingestion

The paper requires exactly-once semantics for critical OLAP use cases: ingestion deduplicates on a primary key, so retried or replicated messages don't double-count. This is what makes financial-grade dashboards possible on an at-least-once transport.

Data freshness

Distinct from query latency (response time of one query). Most Uber use cases need seconds-level freshness; much of the Pinot engineering (async archival, upsert without coordination) defends this SLA.

CAP theorem

The formal reason the paper's seven requirements are jointly unsatisfiable. Uber resolves it per use case: surge pricing chooses freshness/availability over consistency; payments choose the reverse via active-passive consumption.

Upsert

Uber's headline Pinot contribution: the first upsert in an open-source real-time OLAP store. The input stream is partitioned by primary key so all versions of a record land on one node — shared-nothing, no central coordinator, no single point of failure.

Dead letter queue

Open-source Kafka forces a failing consumer to drop messages or retry forever. Uber's DLQ publishes persistently failing messages to a separate topic for later purge or on-demand retry — critical for flows like trip-receipt processing.

Cluster federation

Kafka clusters perform best under ~150 nodes, so Uber federates: producers and consumers see a logical cluster while a metadata server routes to physical ones — enabling transparent topic migration, horizontal growth, and consumer redirection without restarts.

Consumer Proxy

With tens of thousands of consuming apps in four languages, thick Kafka client libraries became unmanageable. The proxy consumes on the app's behalf and dispatches via gRPC push — thin generated clients, per-message retry with DLQ, and parallelism beyond Kafka's partition-count limit.

uReplicator

Replicates topics between regional and aggregate clusters. Its rebalancing algorithm minimizes affected partitions and adaptively sheds bursty load to standby workers. It also checkpoints source→destination offset mappings — the key to active-passive failover.

Chaperone

Collects statistics such as unique message counts per time window from every stage of every Kafka pipeline, compares them, and alerts on mismatch — continuous end-to-end evidence that nothing was lost or duplicated.

FlinkSQL

Built on Apache Calcite: SQL → logical plan → optimized physical plan → Flink job. It made streaming accessible to data scientists, operations and product managers — and made the platform responsible for sizing, scaling and healing jobs its users can't see.

Changelog / CDC

Change Data Capture turns database mutations into events. At Uber, database changelogs are one of the three big Kafka sources, alongside app event data and microservice logs.

Lambda architecture

The classic answer to stream/batch duality. The paper rejects it for maintenance complexity and implementation divergence, but its SQL-based backfill (one query compiled to two Flink jobs) is a deliberately Lambda-shaped compromise with a single user codebase.

Kappa architecture

Eliminates the batch stack but requires weeks-to-months of Kafka retention. Infeasible at Uber's trillions of messages/day with few-day retention — hence Kappa+, which runs unchanged streaming code against the Hive/HDFS archive instead.

Backfill

Because Kafka keeps only days of data, backfill reads from HDFS/Hive. Uber's two mechanisms: compile the same SQL into streaming and batch Flink jobs, or Kappa+ — the streaming code reading bounded historical data with throttling and enlarged out-of-order buffers.

Active-active

Uber's availability-first multi-region pattern (surge pricing): identical Flink jobs in each region consume the same aggregate Kafka feed, so state converges without cross-region replication. Failover = promote another region.

Active-passive

The consistency-first pattern (payments, auditing): a single named consumer reads in the primary region, and uReplicator's checkpointed offset mappings plus an offset sync job let the standby region resume from exactly the right message.

Segment

Pinot tables are stored as segments. Originally each completed segment was synchronously backed up through a single controller (a bottleneck and freshness risk); Uber made archival asynchronous with failed replicas recovering from peer servers instead.

Operator pushdown

Uber extended Presto's planner and Connector API to push predicates, projections, aggregations and limits down into Pinot — the work runs next to Pinot's indices, making sub-second full SQL over fresh data possible.

Avro

A compact, schema'd, row-based format suited to write-heavy landing. Compaction jobs later rewrite Avro into columnar Parquet for efficient analytical scans.

Parquet

Column-oriented storage with compression and predicate-skipping, ideal for analytics. The Avro→Parquet compaction pipeline is the boundary between landing fast and querying fast.

MPP

Presto's execution model: queries are split across a fleet of workers that exchange data in memory without materializing intermediate results, delivering interactive latencies over large datasets.

Watermark

Watermarks let windowed computations close despite out-of-order arrival. Backfilling from archives makes this harder — historical data is far less ordered than the live stream, requiring larger buffering windows in Kappa+.

FAQ

Deliberate strategy, stated in the paper: adopt the strongest open-source system for each layer, re-engineer it for Uber's scale, and contribute the fixes back — leveraging community velocity instead of competing with it. Google (Photon, F1 Lightning, Procella), Facebook (Puma/Swift/Scuba) and Alibaba (Hologres) built proprietary stacks; Uber got comparable capability while uReplicator, Pinot upsert and the Presto pushdown work became community features others now use.

Your trip events land in a regional Kafka cluster and replicate into aggregate clusters in every region. In each region, an identical Flink job consumes that aggregate feed and runs a machine-learning algorithm over time windows, computing a pricing multiplier for each hexagonal geofence. A coordinating service designates one region primary; its update service writes multipliers to an active-active key-value store that the ride-hailing services read. End to end: seconds — and late-arriving events are simply excluded, because a fresh, slightly imperfect price beats a perfect, stale one.

Each occupies a different abstraction layer with different guarantees: Kafka moves events (pub-sub, at-least-once), Flink computes on them continuously (stateful, checkpointed), Pinot answers analytical queries in sub-seconds (indexed, columnar), HDFS remembers everything forever (cheap, consistent archive), and Presto asks flexible questions across all of it (in-memory MPP SQL). The paper's Figure 2 formalizes this as seven layers; remove any system and some layer's guarantee disappears.

Four big things. Federation: physical clusters stay under ~150 nodes (the empirical sweet spot) behind one logical cluster, so scaling and migrations don't touch clients. Dead letter queues: poison messages park in a side topic instead of blocking consumers or being dropped. Consumer Proxy: a middle layer consumes on behalf of apps and pushes via gRPC — thin generated clients in four languages, and parallelism beyond partition count. And uReplicator + Chaperone: audited cross-cluster replication with proof of zero loss.

FlinkSQL. A user writes SQL; the platform (built on Apache Calcite) compiles it through logical and physical plans into a Flink job — 'in production in a span of mere hours.' The catch is operational: users can't reason about the parallelism or memory of jobs they didn't write, so the platform estimates resources from empirical job profiles (stateless jobs are CPU-bound, stream-stream joins memory-bound), auto-scales on load and GC signals, and auto-restarts stuck jobs via a rule engine.

Upsert means updating a record that already arrived — a corrected fare, a delivery status change — while queries keep running at tens of thousands of QPS. The hard part is that all versions of a record must be reconciled somewhere without a central coordinator becoming a bottleneck. Uber's design partitions the input stream by primary key, so every version of a record lands on the same server — shared-nothing, no single point of failure. It was the first upsert support in any open-source real-time OLAP system and is now a flagship Pinot feature.

Two patterns by use case. Active-active (surge pricing): every region independently runs the same Flink job on the same aggregated Kafka feed, so each has the answer ready; a coordinator just promotes a new primary. Active-passive (payments): only one named consumer reads, and because Kafka offsets differ between clusters, uReplicator continuously checkpoints an offset translation table — on failover, the standby resumes from exactly the right message with no loss and no reprocessing storm.

Because streams forget. Kafka retains a few days; testing a new pipeline, training an ML model, or fixing a bug needs months of history that only exists in HDFS/Hive — a different system with different ordering and rate characteristics. Lambda architecture answers with two codebases (drift), Kappa with huge Kafka retention (cost). Uber's compromise: compile one SQL query into both a streaming and a batch Flink job, or run the unchanged streaming code on archived data (Kappa+) with throttling and larger out-of-order buffers.

The Kafka + Flink + Pinot (or Druid/ClickHouse) + object-store + Presto/Trino shape remains the canonical open-source real-time stack — this paper is its reference citation. And the 2021 future-work list predicted the ecosystem's actual trajectory: unified stream/batch processing (Flink's runtime convergence), tiered storage (landed in both Kafka and Pinot), and cloud-agnostic deployment. The systems evolved; the architecture and its trade-off doctrine aged remarkably well.

References

The works the paper builds on — every entry links to the cited paper.

  1. [1]

    Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: a distributed messaging system for log processing. NetDB 2011

  2. [2]

    Carbone, P., Katsifodimos, A., Ewen, S., et al. (2015). Apache Flink: Stream and batch processing in a single engine. IEEE Data Engineering Bulletin 38(4)

  3. [3]

    Im, J.-F., Gopalakrishna, K., Subramaniam, S., et al. (2018). Pinot: Realtime OLAP for 530 million users. SIGMOD 2018

  4. [4]

    Sethi, R., Traverso, M., Sundstrom, D., et al. (2019). Presto: SQL on everything. ICDE 2019

  5. [5]

    Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010). The Hadoop Distributed File System. IEEE MSST 2010

  6. [6]

    Uber Engineering (2016). uReplicator: Uber Engineering's robust Kafka replicator. Uber Engineering Blog / GitHub

  7. [7]

    Uber Engineering (2016). Chaperone: Audit Kafka end-to-end. Uber Engineering Blog

  8. [8]

    Begoli, E., Camacho-Rodríguez, J., Hyde, J., et al. (2018). Apache Calcite: A foundational framework for optimized query processing over heterogeneous data sources. SIGMOD 2018

  9. [9]

    Toshniwal, A., Taneja, S., Shukla, A., et al. (2014). Storm @Twitter. SIGMOD 2014

  10. [10]

    Noghabi, S. A., Paramasivam, K., Pan, Y., et al. (2017). Samza: Stateful scalable stream processing at LinkedIn. VLDB 2017

  11. [11]

    Yang, F., Tschetter, E., Léauté, X., et al. (2014). Druid: A real-time analytical data store. SIGMOD 2014

  12. [12]

    Ananthanarayanan, R., Basker, V., Das, S., et al. (2013). Photon: Fault-tolerant and scalable joining of continuous data streams. SIGMOD 2013

  13. [13]

    Chen, G. J., Wiener, J. L., Iyer, S., et al. (2016). Realtime data processing at Facebook. SIGMOD 2016

  14. [14]

    Chattopadhyay, B., Dutta, P., Liu, W., et al. (2019). Procella: Unifying serving and analytical data at YouTube. VLDB 2019

  15. [15]

    Yang, J., Rae, I., Xu, J., et al. (2020). F1 Lightning: HTAP as a service. VLDB 2020

  16. [16]

    Akidau, T., Bradshaw, R., Chambers, C., et al. (2015). The Dataflow model: A practical approach to balancing correctness, latency, and cost. VLDB 2015

  17. [17]

    Kreps, J. (2014). Questioning the Lambda architecture (the Kappa architecture). O'Reilly Radar

  18. [18]

    Gilbert, S., & Lynch, N. (2002). Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services (CAP). ACM SIGACT News 33(2)

Cite This Paper

@inproceedings{fu2021realtime,
  title     = {Real-time Data Infrastructure at Uber},
  author    = {Fu, Yupeng and Soman, Chinmay},
  booktitle = {Proceedings of the 2021 International Conference on
               Management of Data (SIGMOD '21)},
  year      = {2021},
  publisher = {ACM},
  doi       = {10.1145/3448016.3457552},
  url       = {https://arxiv.org/abs/2104.00087}
}