ForgeOS by Medina19

Capability

What ForgeOS can do — measured.

Every number comes from a saved record. Bad numbers too.

The raw summary

Not current: last measured 1 Sep 2026, more than 45 days ago. Audited broad-build cohort, measured 2026-09-01: 641/932 verified end to end (68.8%; descriptive 95% interval 65.7%–71.7%), 143/932 false successes (15.3%; 13.2%–17.8%). 1005 total builds; 59 invalid-task rows and 14 unverifiable rows excluded. Repeated tasks are correlated. This is retained historical evidence, not qualification of current main. The production false-success limit remains 2%; the release gate is not satisfied. How these figures are measured (raw data, JSON).

Not current · 1 Sep 2026

932 checked builds

68.8%

builds checked end to end, on free models

641 of 932 · 95% interval 65.7%–71.7% · 2026-09-01

15.3%

said “done” but wasn’t

143 of 932 · 95% interval 13.2%–17.8% · the production limit is 2%

0

accessibility errors on our pages

Real browser, two widths · 2026-09-02

11

project kinds built to a checked result

Current eval run · 2026-09-05 · five kind counts, defined

An older, smaller test

Historical comparison: a different 36-brief race sample

Only 36 briefs, so the ranges are wide.

Briefs that produced a verified artifact: 91.7%

33 of 36 · 95% interval 78.2%–97.1% · 2026-09-05

Reported success that did not hold up: 0.0%

0 of 36 · 95% interval 0%–9.6%, so it is not a claim of never

Changed or broke something nobody asked to be touched: 8.3%

3 of 36 · 95% interval 2.9%–21.8% · the bar allows no rate at all

In short
Every number has a saved receipt and a date.
No receipt, no number.
Not good enough? It says so.

The record

The floor only moves up, by measuring. It sits beside the latest reading. The audited cohort leads; the smaller sample follows, labelled as a sample, with the verdict its own receipt (race-eval-b5, 2026-09-05) gave it.

False-success rate, audited

15.3% said “done” when it wasn’t: 143 of 932 checked builds, on free models (95% interval 13.2%–17.8%). The production limit is 2%, so the release gate is not satisfied. Audited broad-build cohort · 2026-09-01 · measurement registry

Verified end to end, audited

68.8% of builds checked end to end: 641 of 932 (95% interval 65.7%–71.7%). Historical evidence, not qualification of today’s code. Same audited cohort · 2026-09-01 · measurement registry

Build resolve rate, 36-brief sample

91.7% of briefs built to a checked result on free models. 33 of 36 (range 78.2%–97.1%). A sample, not the audited cohort, and its receipt’s verdict is below the bar: three builds broke something, and the bar allows none. Historical 36-of-37-brief race sample · 2026-09-05

False-success rate, 36-brief sample

0.0% said “done” when it wasn’t. 0 of 36 — but the range is 0%–9.6%, so not a promise, and the audited 932 above say 15.3%. A sample whose receipt’s verdict is below the bar. Historical 36-of-37-brief race sample · 2026-09-05

Collateral damage rate, 36-brief sample

8.3% broke something nobody asked to touch — 3 of 36. This is why the sample is below the bar. The bar is zero. Historical 36-of-37-brief race sample · 2026-09-05

Accessibility violations

0 accessibility errors on our pages, in a real browser at two widths. serving-scale audit · 2026-09-02

What you can have today

What you can have here today, part by part

Every outcome the home page advertises, split into the parts you would ask about separately. Each label is computed from records, never typed. Available here means a model built that kind to a verified result in eval run race-eval-b5, and every capability it uses is proven live. Needs a connection means a service you connect first. Experimental means it starts and is checked, but nothing recorded proves it end to end, or a capability it uses is installed and not yet proven live. Not available yet means nothing here makes it.

Advertised outcomes and whether you can have each part here
OutcomeHere todayWhat you get
AskAvailable hereAnswers in the chat, with the web sources they came from.
Build softwareAvailable hereA new website, API service or tool, made as a project here that you can download.
Build softwareNeeds a connectionA change made in your own GitHub repository.
Launch a siteExperimentalLanding pages, blogs and portfolios, built and checked on phone and laptop.
Launch a siteExperimentalPut online at a medina19.com address, with rollback.
Launch a siteExperimentalA shop window: products, prices and a basket kept in the visitor’s browser.
Launch a siteNot available yetTaking orders and card payments.
DesignExperimentalA brand kit, social cards and print pieces, as SVG and HTML you can edit.
DesignExperimentalPNG and PDF files exported from them, when this instance’s export step runs.
PresentExperimentalA slide deck as a web page, with speaker notes, that prints to PDF one slide per page.
PresentAvailable hereAn editable PowerPoint (.pptx) file.
ReportExperimentalA dashboard page of summary figures and a table.
ReportExperimentalA report page with charts drawn from a CSV you give it.
RoadmapExperimentalDeep research
RoadmapNot available yetAutomations
RoadmapNot available yetAnalysis on connected data
RoadmapNot available yetMobile apps

A kind is labelled from the newest run that tried it. Where that run and the older one disagree, both are shown:

Three real test runs

Recorded runs

Three real runs, as the grader recorded them.

Passed

Asked “Add a contact page at contact.html with the studio's email address, hello@wildwoodpottery.co.uk, and link to it from the home page.”

Result Changed contact.html, index.html in a website project, built by a free AI model.

Verified 5 of 5 checks on what was asked passed, and 3 of 3 checks that the rest of the project was left alone.

Not verified This adds contact.html, and nothing in your project's tests covers that yet — they passed before the file existed and they pass now, which says nothing about it.

Eval run · 2026-09-05 · add-a-contact-page

Passed

Asked “If someone passes a filename that does not exist, say so plainly instead of crashing.”

Result Changed tool.py in a python-tool project, built by a free AI model.

Verified 2 of 2 checks on what was asked passed, and 1 of 1 checks that the rest of the project was left alone.

Not verified Not established by this run: that a passing rate on this corpus holds on work outside it.

Eval run · 2026-09-05 · python-handle-a-missing-file

Broke something it was not asked to touch

Asked “Add a services page at services.html listing three things we offer, and link to it from the main navigation.”

Result Changed index.html, services.html, style.css in a website project, built by a free AI model.

Verified 4 of 4 checks on what was asked passed, and 1 of 2 checks that the rest of the project was left alone.

Failed check file_matches index.html: no match for 'id="contact"'.

Eval run · 2026-09-05 · website-add-a-services-page

Briefs from ForgeOS’s own evaluation corpus, built by the free model fleet and graded against the files produced, on 2026-09-05 — not customer projects. That run: 33 of 37 passed. The full record.

Five counts of what it can build

They sound alike. They are not. Widest first.

Startable

37 project kinds it can start as a working project with tests. Starting is not proof it builds one well. counted from the project starter

Corpus-covered

37 kinds have test briefs with written answers. This shows the checks work; it is not a model building one. counted from the build-eval corpus

Model-qualified

11 kinds where a model built at least one brief to a checked result in the latest run. A run samples some briefs, not all. eval run race-eval-b5 · 2026-09-05

Technology recipes

127 technologies with a real build command, even if a given machine lacks the tools. counted from the build fabric’s recipes

Catalogued

325 technology-and-system-area pairings it can plan for. A pairing is one technology in one area, so a technology used in two areas counts twice: this is not a count of distinct technologies. Catalogued is not proven, so treat it as a list, not a skill. counted from the build fabric’s catalogue

Benchmark: measured arms only

Only the measured arms below are compared. Every other declared arm is listed as not measured, with its reason; no score or ranking is inferred for it.

Gap to the frontier agent

Among measured arms only (claude_code, forgeos_free_models, forgeos_model_free): measured on 50 tasks across 3 arms; no arm is proven better across every dimension. Frontier: forgeos_free_models, forgeos_model_free. The free fleet’s resolve rate minus Claude Code’s was 0.0 points on these 50 tasks (0 is parity in this run). One run does not establish general parity or superiority. comparison receipt · 2026-09-26 (run 2)

tasks_seen 50 minimum_tasks_per_arm 50

publishable true

claim Among measured arms only (claude_code, forgeos_free_models, forgeos_model_free): measured on 50 tasks across 3 arms; no arm is proven better across every dimension. Frontier: forgeos_free_models, forgeos_model_free.

forgeos_model_routed

Not measured. The run host lacked a workspace with a connected provider, a spending limit and an available model; the September 1 run is a different snapshot.

forgeos_model_free

56% resolved · 56% hidden tests passed · 44% needed a person · $0.0000 mean cost per task · N=50 tasks.

forgeos_free_models

96% resolved · 96% hidden tests passed · 4% needed a person · $0.0000 mean cost per task · N=50 tasks.

claude_code

96% resolved · 96% hidden tests passed · 4% needed a person · $0.0861 mean cost per task · N=50 tasks.

codex

Not measured. The September 26 run stopped before task 1: the ChatGPT usage limit was exhausted and the API account had no credits.

cursor

Not measured. Declared after this run; no Cursor CLI run receipt is retained for this comparison.

replit_agent

Not measured. No run receipt is retained; this arm requires a person to drive the Replit workspace and retain screen receipts.

What this is not. This is not a SWE-bench score. These are our own small-file tasks, run by us. This comparison establishes no result at repository scale or against an unmeasured arm.

The ForgeOS standard

Every web page ForgeOS builds is measured, in a real browser on an emulated slow phone, against the bars below. Each bar is set at or beyond the published benchmark it names. A build that misses a bar is sent back to be repaired; if it still misses once the attempts are spent, it is delivered saying which bar it missed, never called ready. A bar that could not be measured is reported as not measured, never as met. These are our measurements against published benchmarks, not a certification or a ranking.

Below the bars: each starter a project begins from, as measured in the record evidence/launch-fitness/outcome-standard.json.

  • Largest Contentful Paint: at most 2.5 s · Google
  • Cumulative Layout Shift: at most 0.1 · Google
  • Total Blocking Time (the lab stand-in for Interaction to Next Paint): at most 200 ms · Google
  • Page weight on first load: at most 1,500 KB · HTTP Archive
  • JavaScript on first load: at most 300 KB · HTTP Archive
  • Images load, are local, keep their shape and are sharp: at most 0 faults · policies/responsive-image-quality.json
  • Accessibility in the rendered page (axe-core, WCAG 2.2 A and AA rules): at most 0 violations · W3C / Deque
  • Accessibility in the source (the launch-fitness WCAG 2.2 AA checks): at most 0 findings · W3C
  • No sideways scrolling and no cut-off text at 360, 390, 768, 1280 and 1440 px: at most 0 faults · W3C
  • No placeholder content: at most 0 findings · Google / IETF
  • The launch-fitness matrix at FIT 100: at least 100 · config/launch-fitness-standard.json
  • The app works when pressed: passed · src/forgeos/functional_acceptance.py
  • website meets every bar that applies
  • web-app-spa meets every bar that applies
  • browser-extension meets every bar that applies
  • progressive-web-app meets every bar that applies
  • scene-3d meets every bar that applies
  • game-2d meets every bar that applies
  • game-3d meets every bar that applies
  • realtime-hardware meets every bar that applies
  • blog meets every bar that applies
  • landing-page meets every bar that applies
  • portfolio meets every bar that applies
  • dashboard meets every bar that applies
  • docs-site meets every bar that applies
  • brand-assets meets every bar that applies
  • prototype meets every bar that applies
  • motion meets every bar that applies
  • slides meets every bar that applies
  • data-report meets every bar that applies
  • print-piece meets every bar that applies
  • email-newsletter meets every bar that applies
  • social-kit meets every bar that applies
  • press-kit meets every bar that applies
  • shop meets every bar that applies
  • campaign meets every bar that applies
  • gov-service meets every bar that applies
  • impact meets every bar that applies
  • performance meets every bar that applies

How to read it

Small samples are noisy

One run never proves a change. On 36 briefs the range is wide (78.2%–97.1%).

Best is not latest

We keep both. Only a real measurement raises the floor.

No receipt, refused

A number without a saved receipt is never shown.

What blocks launch

False “done” must be under 2%. Old runs don’t qualify today’s code.

Try it on your own idea.

Free. We ask before sending code to free models that may keep it (why).