Teraport Atlas — Calibrated Task Families

Harness-agnostic

Families are self-contained task directories with explicit environments and scoring, built to the contract your harness already executes. No custom runner, no import step.

Parameterized families

A family layer generates seeded variants of every task, in the spirit of the METR Task Standard — fresh instances on demand.

The gauntlet

Static lint, oracle and no-op calibration, determinism replay — run on every version, with the verdict recorded per stage.

The warranty

Every run emits a content-hashed acceptance certificate — including rejections. Acceptance criteria are agreed before authoring, and the certificate is the evidence they are judged against.

Inside the Atlas workspace

Authored, evaluated, and audited in one place: author families with live validation, run evaluation packs across models, and drill from an aggregate score to the exact rollout transcript that produced it.

Every task version rolls up its evidence: per-criterion grades across rollouts, score distribution, trend over time, and pass rate per model.

Per-criterion grade grid across rollouts
Score distribution and daily trend
Pass rate broken down by model

Each rollout is one model against one task version — scored by the task's own verifier, annotated, and traceable down to the transcript.

Model, score, state, and annotation per run
Launch new rollouts against any model
Drill into any run for its full transcript

Token usage and spend are tracked per rollout and rolled up by model and by day — so the price of statistical confidence is always visible.

Spend by model with input/output tokens
Daily spend trend per version
Cost per measured rollout

Space-wide analytics aggregate every rollout: score distributions, rollout volume, cumulative history, and per-model comparisons.

Score distribution across all scored runs
Rollout volume and cumulative history
Average score and pass rate by model

Version summary with score distribution, trend, and pass rate by model

Rollouts table with per-run scores, states, and annotations

Cost analysis with spend by model, daily spend, and cost share

Insights dashboard with score distribution and per-model analytics

Built on open standards and published work

Atlas packages evaluation work in formats your stack can inspect, reproduce, and run without a custom runner. The task family layer builds on existing standards for agent environments, benchmark execution, and rubric-based scoring.

METR Task Standard

The family layer's parameterized tasks follow the spirit of METR's Task Standard, the reference for packaging agentic tasks with explicit environments and scoring.

Open task directories

Atlas families are self-contained task directories: environment, instructions, and scoring live together in one portable contract. If your harness runs standard agentic task directories, it runs Atlas families unchanged.

PaperBench (OpenAI, 2025)

Optional weighted rubrics score partial credit the way PaperBench's rubric trees grade paper reproduction — graded steps instead of all-or-nothing rewards.

Frequently asked questions

Short answers to what evaluation teams usually ask before putting a family in front of their stack.

Do we need to change our harness to run Atlas families?

No. Families are self-contained task directories with explicit environments and scoring. If your evaluation stack executes standard agentic task directories, it executes Atlas families today, with no custom runner and no import step.

What exactly does the acceptance certificate assert?

It asserts what each gauntlet stage returned, against which content hash, at which thresholds: static lint, oracle and no-op calibration, and determinism replay. It records the stages that ran and their verdicts — nothing more. It is emitted for rejections as well as acceptances, so the record is the same whether we passed or failed.

What happens if a family fails our internal review?

Acceptance criteria are agreed in the contract before authoring starts, and the certificate is the evidence they are judged against — portable, machine-checkable, and tied to the exact content hash you evaluated. If a family misses the agreed bar, the remedy is written into the engagement rather than left to argument after delivery.

How do you prevent reward hacking and contamination?

Reference solutions and tests are bind-mounted only at verification time, the verifier log directory is reset before grading, and families regenerate fresh variants per seed — so agents cannot read their grader, forge rewards, or memorize a fixed task instance.

How do we start?

Tell us the capability you need to measure and we scope a calibrated family against it, delivered in your harness. Engagements start with a scoped pilot rather than a self-serve account; workspace access is provisioned per organisation as part of it.

Need an AI engineering partner that can move quickly and build responsibly?

We help US businesses turn AI ambition into working software through product engineering, automation, system design, and delivery support.

Get in touch

Calibrated task families for frontier-model evaluation.