Benchmarks

The rule

No claim without a number, a date and a version. The comparison pages are the most valuable asset this site will have and the easiest to destroy, and one unsupported claim about a named competitor costs more than every page here is worth in traffic.

The rule is enforced by the schemas rather than by review: a comparison entry cannot compile without a competitor version, a measurement date and an anchor into this page, and a benchmark claim cannot compile without its value, its corpus, its run date and the engine build it measured.

The corpus

One set, committed: 30 fixtures in bench/smoke, run on every pull request. They are 15 procedurally generated scenes at 64 and 256 pixels, each rendered in two variants — clean, and Gaussian blur at sigma 0.7, which is roughly a soft rescale.

A scene is a background rectangle plus non-overlapping shapes on a grid: discs, rectangles, rounded rectangles, triangles, regular polygons, stars and a D-shape, in palettes of two to six colours. Nothing overlaps, because an occluded shape has no recoverable boundary and its truth would have to describe what a tracer ought to see rather than what the scene contains.

Ground truth is generated rather than annotated — the palette, the region count, the node count, the primitive list and boundary polylines sampled at 0.01 px all come from the generator, so there is nothing to disagree about. Placement jitter comes from a seeded SplitMix64 stream, so the committed corpus regenerates byte for byte on any machine, and the raster comes from resvg rather than our own renderer, so a rasterizer bug cannot cancel itself out at scoring time.

The metrics

Six numbers per fixture: boundary error (mean, p95 and max, in document pixels), region-count error, palette difference in Oklab under Hungarian matching, node-count ratio, output bytes and wall-clock milliseconds.

Boundary error is the symmetric point-to-polyline distance, measured against the nearest segment rather than the nearest vertex. A triangle edge has no vertex in the middle of it, so a vertex-to-vertex metric reports a perfectly traced triangle as tens of pixels of error; and measuring in one direction only rewards a tracer for emitting extra paths, or for emitting almost nothing.

Node count is reported next to boundary error on purpose. Winning on boundary error by emitting three times the nodes is not winning — it is a file nobody can edit and a cutter stutters through.

The baselines

vtracer and potrace, both open source, both run locally at their own defaults. Tuning another tracer against our own corpus would make the comparison ours rather than theirs. Potrace is bi-level and this corpus is colour, so its numbers read as a note about the corpus rather than a result about potrace.

A fixture-SVG baseline re-measures the ground truth against itself and comes back at 0.004 px on boundary p95, 1.000 on node ratio and zero on region count and palette. That residual is the two-decimal rounding in the SVG writer, and it is the honest floor of the whole method: any boundary figure below about 0.01 px is measurement noise, not quality.

A hosted service is declared in the harness and not wired up: it needs network access and its terms checked first (D13). Outputs from one are for evaluation only, never trained on, and never published without those terms clearly permitting it — an unusable comparison is better than an actionable one.

The gate

Every pull request scores the smoke set against the committed bench/baseline.json and fails on a regression greater than 5% in boundary p95, 5% in region-count error or 10% in node count. All three are one-sided: getting better never fails.

Speed is deliberately not gated. Milliseconds on a shared runner are noise, and a flaky gate is a gate people learn to ignore, so the committed baseline stores a zero for time and any published timing comes from a release build.

A gate that only runs before a release is a gate that gets argued with. Running it on every change makes a regression the problem of whoever caused it, while they still remember what they changed.

What is published here

The run above: vt 0.2.0 on the smoke set, measured 2026-08-27. Boundary error lands under half a pixel at p95, and the outline carries 6.35× the nodes the source geometry has.

That second number is the shipped engine being honest about itself. It wraps VTracer (D02) behind a prepare stage — an exact palette and an upscale of small inputs — which is where the boundary figure comes from; the node count is what the legacy curve fitter charges for it. The week-4 acceptance criterion is beating this run on boundary p95 AND on node count for icons, logos and primitives — both, not either. The figure stays on this page in the meantime, next to the target it misses.

Anyone can reproduce it: the corpus, the ground truth, the baseline and the harness are all in the repository, and docs/benchmarks/methodology.md has the commands.

What is not measured yet

Two more raster variants — JPEG q75, and blur plus JPEG. The engine workspace has a pure-Rust JPEG decoder and no encoder, so they land with the encoder rather than with a fixture nobody can regenerate.

Two larger synthetic sets, declared and not yet generated: standard (around 700 fixtures) and full (around 4,200, on demand). A 1024-pixel size, an icon and clip-art set, and a curated real set of 60 images scored with image-similarity metrics instead of ground truth, gradient-heavy artwork tagged out of scope rather than quietly dropped.

Two more metrics: primitive recovery, and corner precision and recall. And the comparison this page exists for — a head-to-head against a hosted service with side-by-side renders and error heat maps — which is a week-12 milestone. None of these is described above as though it exists, and none of them will be until a run produces a number with a date and a version on it.