The record
The floor only moves up, by measuring. It sits beside the latest reading. The audited cohort leads; the smaller sample follows, labelled as a sample, with the verdict its own receipt (race-eval-b5, 2026-09-05) gave it.
False-success rate, audited
15.3% said “done” when it wasn’t: 143 of 932 checked builds, on free models (95% interval 13.2%–17.8%). The production limit is 2%, so the release gate is not satisfied. Audited broad-build cohort · 2026-09-01 · measurement registry
Verified end to end, audited
68.8% of builds checked end to end: 641 of 932 (95% interval 65.7%–71.7%). Historical evidence, not qualification of today’s code. Same audited cohort · 2026-09-01 · measurement registry
Build resolve rate, 36-brief sample
91.7% of briefs built to a checked result on free models. 33 of 36 (range 78.2%–97.1%). A sample, not the audited cohort, and its receipt’s verdict is below the bar: three builds broke something, and the bar allows none. Historical 36-of-37-brief race sample · 2026-09-05
False-success rate, 36-brief sample
0.0% said “done” when it wasn’t. 0 of 36 — but the range is 0%–9.6%, so not a promise, and the audited 932 above say 15.3%. A sample whose receipt’s verdict is below the bar. Historical 36-of-37-brief race sample · 2026-09-05
Collateral damage rate, 36-brief sample
8.3% broke something nobody asked to touch — 3 of 36. This is why the sample is below the bar. The bar is zero. Historical 36-of-37-brief race sample · 2026-09-05
Accessibility violations
0 accessibility errors on our pages, in a real browser at two widths. serving-scale audit · 2026-09-02
What you can have today
What you can have here today, part by part
Every outcome the home page advertises, split into the parts you would ask about separately. Each label is computed from records, never typed. Available here means a model built that kind to a verified result in eval run race-eval-b5, and every capability it uses is proven live. Needs a connection means a service you connect first. Experimental means it starts and is checked, but nothing recorded proves it end to end, or a capability it uses is installed and not yet proven live. Not available yet means nothing here makes it.
A kind is labelled from the newest run that tried it. Where that run and the older one disagree, both are shown:
Three real test runs
Recorded runs
Three real runs, as the grader recorded them.
Passed
Asked “Add a contact page at contact.html with the studio's email address, hello@wildwoodpottery.co.uk, and link to it from the home page.”
Result Changed contact.html, index.html in a website project, built by a free AI model.
Verified 5 of 5 checks on what was asked passed, and 3 of 3 checks that the rest of the project was left alone.
Not verified This adds contact.html, and nothing in your project's tests covers that yet — they passed before the file existed and they pass now, which says nothing about it.
Eval run · 2026-09-05 · add-a-contact-page
Passed
Asked “If someone passes a filename that does not exist, say so plainly instead of crashing.”
Result Changed tool.py in a python-tool project, built by a free AI model.
Verified 2 of 2 checks on what was asked passed, and 1 of 1 checks that the rest of the project was left alone.
Not verified Not established by this run: that a passing rate on this corpus holds on work outside it.
Eval run · 2026-09-05 · python-handle-a-missing-file
Broke something it was not asked to touch
Asked “Add a services page at services.html listing three things we offer, and link to it from the main navigation.”
Result Changed index.html, services.html, style.css in a website project, built by a free AI model.
Verified 4 of 4 checks on what was asked passed, and 1 of 2 checks that the rest of the project was left alone.
Failed check file_matches index.html: no match for 'id="contact"'.
Eval run · 2026-09-05 · website-add-a-services-page
Briefs from ForgeOS’s own evaluation corpus, built by the free model fleet and graded against the files produced, on 2026-09-05 — not customer projects. That run: 33 of 37 passed. The full record.
Five counts of what it can build
They sound alike. They are not. Widest first.
Startable
37 project kinds it can start as a working project with tests. Starting is not proof it builds one well. counted from the project starter
Corpus-covered
37 kinds have test briefs with written answers. This shows the checks work; it is not a model building one. counted from the build-eval corpus
Model-qualified
11 kinds where a model built at least one brief to a checked result in the latest run. A run samples some briefs, not all. eval run race-eval-b5 · 2026-09-05
Technology recipes
127 technologies with a real build command, even if a given machine lacks the tools. counted from the build fabric’s recipes
Catalogued
325 technology-and-system-area pairings it can plan for. A pairing is one technology in one area, so a technology used in two areas counts twice: this is not a count of distinct technologies. Catalogued is not proven, so treat it as a list, not a skill. counted from the build fabric’s catalogue
Benchmark: measured arms only
Only the measured arms below are compared. Every other declared arm is listed as not measured, with its reason; no score or ranking is inferred for it.
Gap to the frontier agent
Among measured arms only (claude_code, forgeos_free_models, forgeos_model_free): measured on 50 tasks across 3 arms; no arm is proven better across every dimension. Frontier: forgeos_free_models, forgeos_model_free. The free fleet’s resolve rate minus Claude Code’s was 0.0 points on these 50 tasks (0 is parity in this run). One run does not establish general parity or superiority. comparison receipt · 2026-09-26 (run 2)
tasks_seen 50 minimum_tasks_per_arm 50
publishable true
claim Among measured arms only (claude_code, forgeos_free_models, forgeos_model_free): measured on 50 tasks across 3 arms; no arm is proven better across every dimension. Frontier: forgeos_free_models, forgeos_model_free.
forgeos_model_routed
Not measured. The run host lacked a workspace with a connected provider, a spending limit and an available model; the September 1 run is a different snapshot.
forgeos_model_free
56% resolved · 56% hidden tests passed · 44% needed a person · $0.0000 mean cost per task · N=50 tasks.
forgeos_free_models
96% resolved · 96% hidden tests passed · 4% needed a person · $0.0000 mean cost per task · N=50 tasks.
claude_code
96% resolved · 96% hidden tests passed · 4% needed a person · $0.0861 mean cost per task · N=50 tasks.
codex
Not measured. The September 26 run stopped before task 1: the ChatGPT usage limit was exhausted and the API account had no credits.
cursor
Not measured. Declared after this run; no Cursor CLI run receipt is retained for this comparison.
replit_agent
Not measured. No run receipt is retained; this arm requires a person to drive the Replit workspace and retain screen receipts.
What this is not. This is not a SWE-bench score. These are our own small-file tasks, run by us. This comparison establishes no result at repository scale or against an unmeasured arm.
The ForgeOS standard
Every web page ForgeOS builds is measured, in a real browser on an emulated slow phone, against the bars below. Each bar is set at or beyond the published benchmark it names. A build that misses a bar is sent back to be repaired; if it still misses once the attempts are spent, it is delivered saying which bar it missed, never called ready. A bar that could not be measured is reported as not measured, never as met. These are our measurements against published benchmarks, not a certification or a ranking.
Below the bars: each starter a project begins from, as measured in the record evidence/launch-fitness/outcome-standard.json.
- Largest Contentful Paint: at most 2.5 s · Google
- Cumulative Layout Shift: at most 0.1 · Google
- Total Blocking Time (the lab stand-in for Interaction to Next Paint): at most 200 ms · Google
- Page weight on first load: at most 1,500 KB · HTTP Archive
- JavaScript on first load: at most 300 KB · HTTP Archive
- Images load, are local, keep their shape and are sharp: at most 0 faults · policies/responsive-image-quality.json
- Accessibility in the rendered page (axe-core, WCAG 2.2 A and AA rules): at most 0 violations · W3C / Deque
- Accessibility in the source (the launch-fitness WCAG 2.2 AA checks): at most 0 findings · W3C
- No sideways scrolling and no cut-off text at 360, 390, 768, 1280 and 1440 px: at most 0 faults · W3C
- No placeholder content: at most 0 findings · Google / IETF
- The launch-fitness matrix at FIT 100: at least 100 · config/launch-fitness-standard.json
- The app works when pressed: passed · src/forgeos/functional_acceptance.py
- website meets every bar that applies
- web-app-spa meets every bar that applies
- browser-extension meets every bar that applies
- progressive-web-app meets every bar that applies
- scene-3d meets every bar that applies
- game-2d meets every bar that applies
- game-3d meets every bar that applies
- realtime-hardware meets every bar that applies
- blog meets every bar that applies
- landing-page meets every bar that applies
- portfolio meets every bar that applies
- dashboard meets every bar that applies
- docs-site meets every bar that applies
- brand-assets meets every bar that applies
- prototype meets every bar that applies
- motion meets every bar that applies
- slides meets every bar that applies
- data-report meets every bar that applies
- print-piece meets every bar that applies
- email-newsletter meets every bar that applies
- social-kit meets every bar that applies
- press-kit meets every bar that applies
- shop meets every bar that applies
- campaign meets every bar that applies
- gov-service meets every bar that applies
- impact meets every bar that applies
- performance meets every bar that applies
How to read it
Small samples are noisy
One run never proves a change. On 36 briefs the range is wide (78.2%–97.1%).
Best is not latest
We keep both. Only a real measurement raises the floor.
No receipt, refused
A number without a saved receipt is never shown.
What blocks launch
False “done” must be under 2%. Old runs don’t qualify today’s code.
Try it on your own idea.
Free. We ask before sending code to free models that may keep it (why).