EatBetter
Twenty-five days of contract AI engineering on a nutrition app, where every change to the model, the catalogue or the assistant had to be earned by a measurement.
Measure first, then change
When the engagement started there was no versioned way to measure the production meal analysis against the company's own data. Whether one model or prompt was better than another could not be said with evidence. The food catalogue had been filled by model-generated writes, and the path that let defects in was still open.
Twenty-five days later there was a measurement platform anyone with the repository can run, a cleaned catalogue with its source closed, a pipeline that records and judges user corrections, and a fine-tuning dataset that matches the live call exactly. This page describes how each was built and what the measurements said, including the ones that said stop.
From production sample to release decision
Labels are proposed by a three-model panel and settled by a human; model output is never treated as ground truth, and a model that helped propose labels cannot be evaluated as a candidate. Every comparison is paired on the same cases and resampled by user, because one person's meals are not independent observations.
What the platform was used to decide
ΔF1 against the live model, 95% CI [−0.023, +0.012], at about seven times the price. Retired.
Comparisons lost to the bare model, at 12× the cost and 7× the latency. Shut down.
Share of the best prompt variant's apparent gain that came from label granularity rather than better recognition.
Best agreement among five judge candidates on 80 hand-labelled meals; the existing rule path scored 0.66.
An error map drawn in the second week showed that two thirds of the best prompt's gain was not a vision improvement at all. That finding is what separates a higher score from a better system, and it changed how every later target was written.
One permitted path to production
Only a human connected to the live database. Each change was planned locally from a read-only export, rehearsed inside a transaction that was rolled back, and committed only when the affected row count matched the prediction. The write gate at the source went live before the cleanup, so cleaned rows could not return.
What changed in the live database
Percentages are of the 387,668-row catalogue export taken before the cleanup. Deletion went by content group, not by a target number: one-off compositions, numbered copies and broken slugs were removed, while searchable generic names were only unpublished.
23.0% of the catalogue, each row recoverable from its undo file.
23.4%, kept in the database but taken out of publication.
75.4%: model-invented popularity replaced with real usage.
Row operations over three days: chains flattened, wrong merges split, junk brands cleared.
On 21 September the first version of a duplicate-merge script wrote one of two merge fields and broke 1,845 redirects. It was repaired the same day from current live values, verified with a health query, and the script gained a root-quality filter. One other step skipped its separate rollback rehearsal; the count matched the prediction exactly and the deviation was logged. Both are in the engagement's records, which is where they belong.
A scorecard built from real failures
The in-app assistant keeps a memory of each user. In a stress test that adds more than thirty synthetic memory entries to a real profile, an older allergy record stopped reaching the answer. A pre-registered scorecard measured it: 400 frozen cases, 6,000 generations and 24,000 judge slots across two judge models and two orderings.
A fix that places allergy, medication and condition records first in the prompt cut the paired veto rate from 96.4% to 2.1% over 172 pairs (Holm-corrected p = 0.0001). It was the only one of ten candidates to pass the acceptance gate, and was delivered behind a flag that switches on without a deploy.
The lesson came the day before. A 24-variant tournament found no verified improvement, because the control arm already scored 97–98% and the test set never stressed the real mechanism. The scorecard that found the win was built only from cases the current version actually got wrong, at about a quarter of the cost. Safety results are advisory: there was no dietitian on the team.
Data and pipelines delivered alongside
Foods checked independently by three models. The 215,584-food correction package carries a per-row “write only if unchanged” guard and undo SQL, and was rehearsed on a local copy of the live export.
Persistent, budget-capped jobs over a three-endpoint pool with AIMD rate control, at the ceiling of the provider quota; about 250k food images (estimate).
Versioned records for Qwen3-VL with a leakage audit against sealed sets, PII scan, deduplication and per-user splits; 0 template or tokenizer errors over 638,493 records.
Synthetic assistant conversations in six languages covering all 15 tools; 89% passed every code check. No real user chats were used.
Real meal notes labelled through the live contract, after a six-arm teacher benchmark weighed label quality against cost.
Lost user corrections are now recorded and the model's first answer preserved; personal habits and a spend-capped LLM judge sit behind flags.
At handover on 1 October the nutrient package was approved and waiting for an off-peak write, the safety fix was waiting for its flag, and the image total was an estimate still to be confirmed from the job table.
Stopped on evidence
Work that a measurement or a review ended is listed with the same weight as work that shipped. Most of it set the direction of what came next.
- Candidate recognition pipeline. Worse than the bare model in four of four comparisons, at 12× the cost. Closed.
- Prompt-module set. A measured +0.04 F1, significant in three repeats, was judged too small to be visible to users. Archived without a pull request.
- Costlier teacher candidate. 9% worse than the live model at 20× the price. The live baseline stayed.
- Corrections loop, first design. 20 commits and 724 tests, rejected in review: the personal hint read only the note, while half of all scans were photos. Rebuilt as three small, conflict-free pull requests that merged two days later.
- 24-variant tournament. No verified improvement. Its measurement harness was reused for the scorecard that found one.
One engineer, many agents, fixed rules
The work was done by one engineer directing several AI coding agents in parallel. Decisions, rules and every step that touched production stayed with the human; agents worked from 94 written task specifications with limited permissions, each in its own worktree.
- Only a human writes to the live database: rehearse with ROLLBACK, then COMMIT, always with an undo file.
- Behaviour changes sit behind a flag whose default is today's behaviour, so rollback needs no deploy.
- Pull requests are conflict-free by design, checked with a merge-tree matrix against the main branches and each other.
- Adversarial multi-agent review before a pull request opens: 44 agents on one change produced 19 verified findings.
- Mutation testing, pre-registration, Holm correction and user-clustered bootstrap for anything claimed as a result.
The handover was a set of decision records, daily notes and a delivery manifest listing each artefact with its SHA-256.
Client work. The repositories and data are private, so nothing on this page links out, and product internals are described only as far as the method needs. Figures are transcribed from the engagement's own records as of 30 September 2026; items delivered but not yet live at handover are stated as such.