The numbers on the Benchmark tab are measured, not asserted. This page documents exactly how they are produced, so anyone can reproduce them at the same task-set version and seed. No numbers are hardcoded here — the live figures are on the Benchmark tab.
The current task set is task_set_version v2
(eval/task_sets/v2.json). It is content-pinned: every case's canonical
JSON is hashed into the manifest, and the loader verifies each pinned case against the
corpus before a run can start. Editing a pinned case without a deliberate version bump
(with re-pinned hashes) refuses to load — the task set never re-baselines silently.
That rule lives in app/scoreboard.py (load_task_set).
Each class in v2 carries four pinned cases: code_fix,
code_gen, extract, frontend,
reasoning, sql and tool_use.
Every measurement runs two arms on the same task set, at the same seed (seed 7):
Comparing the two at the same version and seed is what makes the scoreboard a comparison instead of a claim.
The live numbers, per arm and task class, are published on the Benchmark tab of the Plans page. If no measurement has been published yet, that page says so instead of showing placeholder numbers.
Run the scoreboard runner against the pinned task set. Every non-dry run needs a real SID key; the run's cost meters on that key.
python scripts/run_scoreboard.py \ --manifest eval/task_sets/v2.json \ --cases eval/capability_cases.jsonl \ --arms "fixed-model=<pinned-open-weight-model>,auto" \ --base-url https://superintelligencedog.com/v1 \ --api-key "$SID_API_KEY" \ --out eval/reports/scoreboard.json
The runner refuses to silently overwrite an existing report; pass
--force only for a deliberate re-measure. Use
--archive-out to keep an immutable timestamped copy of a measurement.