Flakiness.io represents facts about test runs as interval unions over numeric test IDs. This keeps the query index compact and turns dashboard filtering into set algebra.
Flakiness.io uses a purpose-built analytics engine to store and process test data unusually efficiently on modest hardware.
Why does efficiency matter?
Storing test data at scale can be expensive — see one of our estimates.
To see how it works, consider this example with 1,000 tests:
import { test, expect } from '@playwright/test';
for (let testName = 1; testName <= 1000; ++testName) {
test(`the test #${testName}`, async () => {
expect(testName).not.toBe(500);
});
}Notice that all of them pass except test #500, which fails.
Storing test results
Every incoming report is added to the Test Results Index, which powers the dashboard and answers analytics queries.
The index keeps track of every test the project has reported. When a previously unseen test arrives, it is assigned a sequential identifier:
| Test Name | Test ID |
|---|---|
the test #1 | 1 |
the test #2 | 2 |
| … | … |
the test #500 | 500 |
| … | … |
the test #1000 | 1000 |
Next, the report itself is added to the index with the following shape:
{
name: 'run #1',
passed: [1, 499, 501, 1000], // tests 1..499 and 501..1000
failed: [500, 500], // test 500
}Notice the compact arrays of numbers in the passed and failed fields. These arrays are interval unions: each one encodes a sorted collection of non-overlapping, non-adjacent closed intervals as a flat sequence of endpoint pairs. An interval union with intervals therefore contains exactly numbers. For example, [1,8,10,10,15,100] encodes the numbers 1..8, 10, 15..100.
Test runners discover tests in a stable order, so related test IDs tend to be adjacent. Since most tests pass, real outcome sets usually contain long ranges with only occasional holes.
Querying test results
Interval unions are also at the heart of the engine’s query processing.
Flakiness.io provides the Flakiness Query Language — a small language for filtering tests by their attributes. The Test Results Index stores the facts needed to answer these queries: outcomes, tags, annotations, duration buckets, and so on. All of these are represented as interval unions.
Interval unions support typical set operations like union, intersection and subtraction. For example:
The engine then translates FQL queries into a sequence of set operations over interval unions. For example, a search for a word in a test title becomes an intersection between the current result set and the set of test IDs whose names contain the term.
The engine spends most of its time performing set operations over interval unions. A dedicated benchmark environment mirrors our production hardware and measures every operation on synthetic and real-world data.
These benchmarks reveal patterns in real-world data that we can exploit. One example is unionAll, which computes the union of interval unions containing intervals each. The standard heap-based approach runs in . In practice, however, large unions often collapse to a result containing only intervals, where is small. For those workloads, the engine uses a specialized implementation that is faster despite its worse asymptotic complexity.
We just saw how the outcomes of 1,000 synthetic tests fit into a few small arrays.
In the real world, as of August 25, 2026, WordPress’s Gutenberg project has uploaded 188,000 reports containing 420 million test results. The corresponding Test Results Index occupies less than 300 MB, and dashboard queries operate directly on it.
Keeping the index compact and cheap to query lets us keep prices low and provide free test analytics for large open-source projects ❤️
Flakiness.io Test analytics for GitHub & GitLabFlakiness.io Team