Every Wikipedia edit, live, with each estimate checked against an exact count

Transport and resume

The page opens one EventSource on Wikimedia's recentchange stream. Every message carries an SSE id: a JSON list of Kafka topic and partition positions (in the stream as observed, a millisecond timestamp for the active datacenter's topic). Wikimedia closes every connection after 15 minutes. The browser then reconnects on its own and sends the last id it saw as the Last-Event-ID header, so the stream resumes where it stopped. If the browser gives up (an HTTP error it will not retry), the page reconnects itself with since set 5 seconds before the newest event time, which can replay a few events.

Each event's meta.id passes through a set of the 5,000 most recent ids; repeats are dropped and counted in the tape footer. Wikimedia's synthetic canary events (meta.domain = canary) are discarded before anything is counted. Changing the scope resets every counter and sketch.

Edit river

The river draws events of type edit and new from the chosen scope; category changes and log actions appear in the cards. Rows are the 6 wikis with the most edits since the counters started, plus one row for the rest. A wiki takes a row, or passes the row above, only when its count exceeds that row's by 10% plus one, so rows settle without flapping, and they slide to new places. Horizontal position is age, this computer's clock minus meta.dt, so it includes the clock offset in the delay tile. The span grows from 4 to 60 seconds over the first minute. Vertical position inside a row comes from a hash of the event id, so dots do not stack. The river keeps the latest 2,000 edits.

Hashing

All three sketches hash with MurmurHash3, x86 32 bit, over UTF-8 bytes. The implementation is checked in the test suite against the published test vectors and SMHasher's verification value. Count-Min uses a different seed for each row.

HyperLogLog

Distinct editors are counted by user name with p = 12, so m = 4,096 registers of one byte each. The top 12 bits of the hash choose a register, which keeps the largest rank it has seen; the rank is one plus the number of leading zeros in the remaining 20 bits.

Below 2.5 m = 10,240 the linear counting branch is in use, and its error there is a little smaller than the raw bound (close to 1 / √(2m) ≈ 1.1% for small counts). The chart follows one sketch over time, so neighbouring points are strongly correlated; a coverage test would need many independent sketches.

The exact set beside it stores only a 53 bit hash of each user name (two seeded MurmurHash3 values). The chance of any collision among 100,000 editors is about n² / 2^54 ≈ 6 × 10^−7, so it is exact in practice.

Count-Min

Events per wiki go into a table of 4 rows by 1,024 counters. Each event adds one to a counter in every row and the estimate is the minimum over rows. The page uses the standard update, so the textbook bound applies as stated; the conservative update tightens estimates but falls outside that proof.

The proof assumes pairwise independent hash functions, which seeded MurmurHash3 is not provably. With a few hundred wikis in 1,024 columns, a key is inflated only when it collides in all four rows, so most estimates come out exact and the bound is far from tight here. The card puts the share of inflated keys expected under uniform hashing next to the share observed.

Space-Saving

Pages (edits and creations) are tracked with k = 200 counters, after Metwally, Agrawal and El Abbadi (2005).

Every second the page checks both guarantees against an exact map of every page seen and counts any failure. The guarantees are deterministic, so that count should stay at zero; it is on the page so that a bug would show. The ranking is weaker: for pages near N / k, most of a counter's count is error inherited from the page it evicted, so the order of the lower rows is unreliable. The card counts how many of the 15 rows belong in the exact top 15.

Bursts

μ and σ are taken before second t is added, and flagged seconds still update the baseline. Nothing is flagged during the first 30 seconds. Seconds are binned by event time (meta.dt) because delivery comes in clumps, and a clump of delivery is no burst of editing; the card counts what the same rule flags on arrival time bins for comparison. A second is closed, and judged, once the newest event time seen is 2 seconds past it, so the judgement does not depend on this computer's clock. Events that arrive later still fill the chart, miss the detector, and are counted. The rate tile uses the same closed seconds, smoothed with a 10 second half life.

Delay, bot flag and privacy

The delay is arrival time on this computer minus meta.dt from Wikimedia, over the last 2,000 events. It includes the offset between the two clocks, which can be a second or more and can even make the delay negative. The bot flag is the wiki's own: accounts with the bot right. Automated edits from accounts without it count as people.

User names and IP addresses are never displayed. A name is hashed on arrival, into the HyperLogLog registers and the 53 bit exact set, and the string is not kept. The tape and the river show pages and leave out editors. Pages in the User and User talk namespaces (2 and 3) are named after an account or an IP address, so their titles are counted by hash and shown only as “User page (name hidden)”, without a link. Titles in other namespaces can still mention an account, for example a noticeboard subpage; those are shown as the wiki published them.

What the comparison shows

The exact counters are only affordable because this stream is small, tens of events a second as the rate tile shows. The exact set and maps grow with every new editor and page (the page resets itself if the page map reaches 500,000 keys), while the sketches stay at 4 KB, 4 × 1,024 counters and 200 counters. At a scale where exact counting does not fit in memory, the sketch is all there is; running both here shows what the sketch gives up in exchange.