benchfck
A generator and exact harness for machine-state tasks, built so that nothing in the scoring path depends on a learned judge.
cl100k_base for BPE measurementA benchmark that cannot be gamed by its own judge
Most reasoning benchmarks ship a fixed dataset and score answers with another model. Both choices leak: the dataset ages into the training corpus, and the judge brings its own errors. benchfck removes both. It generates every item from a seed, and every step from execution to scoring is exact arithmetic on a pinned virtual machine.
The cost of that choice is scope. It measures controlled machine-state reasoning, and nothing else. It does not claim to measure general capability.
From seed to score
A candidate program is only accepted if the typed intermediate representation and the encodings agree on every one of the 256 possible input bindings. Anything that disagrees is discarded before a model ever sees it.
Four rungs of the same program
Each accepted program is rendered into four instruction encodings. They describe identical behaviour with very different surface forms, which separates a model that reasons about machine state from one that pattern-matches familiar Brainfuck idioms.
- E0 — canonical Brainfuck with implicit pointer-relative addressing.
- E1 — per-item symbol permutation plus an operational legend, so memorised glyphs stop helping.
- E2 — compact explicit operations with a run-length carrier, 3.356× the token count of E0.
- E3 — verbose explicit operations, 7.072× E0.
What a generated item has to clear
Population-level gates are published before any model is run, so the acceptance criteria cannot be tuned after seeing results.
Execution has to actually do work, not idle through steps.
Small input changes must propagate into the machine state.
Programs that collapse into memorised idioms are rejected.
Unique semantic functions identified across 51 coarse profile buckets.
Property tests over a 10,000-program population
Four balanced, non-overlapping CI jobs partition a 10,000-program population, so a regression in one structural family cannot hide behind the others. GitHub Actions runs CI and CodeQL; local control scripts reproduce the same checks offline.
This is a v0.4.0-alpha engineering candidate. No model results have been produced and no leaderboard exists. The release scope is arity 1, with arity 2 deferred to v0.5. The public generator reveals its constructor family by design, so the eight public constructors are a narrow subset of the identified space, and the private scoring epoch has not been activated.