Project · Real-time data pipeline

Wikipedia Data Streaming using Apache Kafka

A real-time data pipeline for processing Wikipedia edits with Apache Kafka. This project walks through building a producer that captures article changes, consumers that enrich and aggregate the events, and a live dashboard that displays the results. It also provides metrics to monitor activity, measure processing delay, and compare activity across language editions.

Explore the pipeline ↗

The data

Real Wikipedia edits, streamed as they happen

The data comes from Wikimedia EventStreams, a public feed of changes across Wikimedia's projects. This project follows edits and new articles on Wikipedia's language editions, made by both people and bots.

What each edit contains

  • The article and its language edition, such as an English or Japanese Wikipedia page.
  • When it happened, with an event ID and a timestamp from Wikimedia.
  • The kind of change, an edit to an existing article or the creation of a new one.
  • The bot flag and article size before and after the change.

The stream describes changes to articles. Article text, editor identities and edit summaries are left out of this project's stored events.

What this project adds

The edits and their source timestamps are real. The pipeline adds the information needed to follow and understand the stream:

  • · The number of bytes added or removed
  • · Kafka partition and offset for tracing each sampled event
  • · Arrival and processing timestamps to measure delay
  • · Activity by language and pages with editing bursts
  • · A sampled live feed and a recorded Kafka replay

The live display refreshes in five-second batches. Its visible edits are sampled; the counters include all accepted events.

Explore the data source ↗View the event schema ↗Wikimedia EventStreams · Wikipedia article changes
Recorded Kafka runCaptured 2026-10-27 · replaying recorded events

Article edits observed

6,190Pipeline total at the time of recording

Human / bot

65% / 35%Using Wikimedia’s bot flag

Source → storage

10.0 secMeasured during the recorded Kafka run

The world is editing

The edit stream

6 on screen
Open an article on Wikipedia ↗
Sampled article edits · select an article to open WikipediaRecorded source data

The stream’s pulse

Recent edit activity

Event-time minute buckets; events over one minute late remain in totals. Tallies cover accepted Wikipedia article edits.

Data architecture

From Wikipedia to the live dashboard

A producer captures article edits, Kafka buffers the events, and two consumers prepare them for the dashboard. The website reads a small, recent view from Supabase.

Wikipedia streaming data architectureIcon-based architecture, grouped into data source, data pipeline and serving. Wikimedia edits flow through a producer, a raw Kafka topic, an enrichment consumer, an enriched Kafka topic and a display consumer. The display consumer writes snapshots to Supabase, which serves the Vercel dashboard through a cached API. Rejected records go to a separate dead-letter Kafka topic.
  1. Data source

    Wikimedia EventStreams

    Wikipedia edits

    Public article-change stream

  2. Data pipeline

    Render

    Producer

    Capture, validate and trim

  3. Aiven · Apache Kafka

    Raw events

    wiki.edits.raw

  4. Render

    Enrichment consumer

    Language and byte change

  5. Aiven · Apache Kafka

    Enriched events

    wiki.edits.enriched

  6. Render

    Display consumer

    Aggregate and sample

  7. Serving

    Postgres

    Supabase

    Bounded frames and metrics

  8. Vercel · Next.js

    Live dashboard

    Cached API · 5-second refresh

  9. Rejected source or raw events go to the separate wiki.edits.dlq Kafka topic.
Illustrative data flowAll Kafka topics share one Aiven service. The producer and consumers run in one Render container.

Capture and buffer

The producer selects Wikipedia article changes and keeps only the fields this project needs. Kafka stores raw and enriched events in separate topics, allowing capture and processing to run independently.

Enrich and aggregate

One consumer adds language and article-size changes. The display consumer builds activity counts, editing-burst candidates and sampled feed batches. It saves its state before acknowledging Kafka offsets.

Store and serve a small view

Supabase keeps a ten-minute ring of feed frames and 24 hours of minute summaries. Vercel serves a cached API response every five seconds, and the browser gradually displays the sampled edits.

Source: Wikimedia EventStreams. Article titles link to their source pages. This independent project is not affiliated with Wikimedia. Language editions identify wikis, not editor locations. The visible feed is a sample.