Get Started

1. Install the SDK

Python
TypeScript

pip install traceroot

npm install @traceroot-ai/traceroot

2. Define a dataset

A Dataset holds your cases. Each case has an input and an optional expected answer. Set key explicitly on any dataset you’ll re-run — it’s what identifies the dataset, so you can rename it later without starting a new history.

Python
TypeScript

from traceroot import Dataset, evaluate

ds = Dataset("Capitals QA", key="capitals-qa")
ds.add({"country": "France"}, expected="Paris")
ds.add({"country": "Japan"}, expected="Tokyo")

import { Dataset, evaluate } from "@traceroot-ai/traceroot";

const ds = new Dataset("Capitals QA", null, { key: "capitals-qa" });
ds.add({ country: "France" }, { expected: "Paris" });
ds.add({ country: "Japan" }, { expected: "Tokyo" });

3. Write a task

The task is the system under evaluation. It maps an input to an output, and it may be async.

Python
TypeScript

CAPITALS = {"France": "Paris", "Japan": "Tokyo"}

def task(input):
    return CAPITALS.get(input["country"], "")

const CAPITALS: Record<string, string> = { France: "Paris", Japan: "Tokyo" };

function task(input: { country: string }) {
  return CAPITALS[input.country] ?? "";
}

4. Write a scorer

A scorer receives the case’s input, output, expected, and metadata, and returns a value. The metric takes the function’s name.

Python
TypeScript

def matches_expected(input, output, expected=None):
    return output == expected

import type { ScorerContext } from "@traceroot-ai/traceroot";

function matchesExpected({ output, expected }: ScorerContext) {
  return output === expected;
}

5. Run it

local=True runs the eval in full and reports nowhere, so you can try it without an API key.

Python
TypeScript

result = evaluate(
    name="capitals",
    dataset=ds,
    task=task,
    scorers=[matches_expected],
    candidate_version="capitals-v1",
    local=True,
)
print(result.summary())

const result = await evaluate({
  name: "capitals",
  dataset: ds,
  task,
  scorers: [matchesExpected],
  candidateVersion: "capitals-v1",
  local: true,
});
console.log(result.summary());

Drop local and the run reports to TraceRoot, using the same credentials as the tracing SDK:

.env

TRACEROOT_API_KEY=your-api-key
TRACEROOT_HOST_URL=https://app.traceroot.ai  # or your self-hosted URL

candidate_version labels what you’re testing — a model id, a prompt version, or a git sha. It’s what the UI groups and compares runs by.

Next steps

Datasets

Author, version, and publish datasets.

Scorers

Code scorers and LLM-as-a-judge.

Running Evals

Every option of evaluate().

Reading Results

Per-metric summaries and per-case results.

Run the example

Clone the repo and run a complete tool-agent eval end-to-end.

Python

Run the Python example

TypeScript

Run the TypeScript example

Overview

Tracing

AI Agents

Detectors

Evaluations

CLI

1. Install the SDK

2. Define a dataset

3. Write a task

4. Write a scorer

5. Run it

Next steps

Datasets

Scorers

Running Evals

Reading Results

Run the example

Python

TypeScript

​1. Install the SDK

​2. Define a dataset

​3. Write a task

​4. Write a scorer

​5. Run it

​Next steps

Datasets

Scorers

Running Evals

Reading Results

​Run the example

Python

TypeScript

1. Install the SDK

2. Define a dataset

3. Write a task

4. Write a scorer

5. Run it

Next steps

Run the example