# traceroot > This file is very large and may be truncated. For an index of this content, please refer to https://traceroot.ai/docs/llms.txt instead. > The open-source platform that continuously improves your ai agents. > ## Agent Instructions > To set up traceroot in a repository, follow the setup prompt in https://traceroot.ai/docs/getting-started/coding-agents.md. # Bring Your Own Key (BYOK) Source: https://traceroot.ai/docs/ai-agent/byok Use your own LLM API keys to power the traceroot Agent By default, traceroot provides system models for the traceroot Agent. With BYOK, you can bring your own API keys to control which model is used. ## Default Models | Provider | Models (first entry is the default) | | - | - | | Anthropic | `claude-opus-4-8`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-sonnet-5`, `claude-sonnet-5-5`, `claude-sonnet-4-6`, `claude-opus-4-5`, `claude-sonnet-4-5`, `claude-haiku-5-5`, `claude-haiku-4-5` | | OpenAI | `gpt-5.5`, `gpt-6-luna`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.4-nano`, `gpt-5`, `gpt-5-mini`, `o3`, `gpt-5.3-codex`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-6-sol` | See [Supported Models](/docs/integrations/internal-models) for a description of each system model. ## Supported Providers OpenAI, Anthropic, DeepSeek, xAI, Moonshot (Kimi), Z.AI (GLM), Google Gemini, OpenRouter, Azure OpenAI, AWS Bedrock, TypeSafe AI. ## BYOK model catalog | Provider | Models | | - | - | | OpenAI | `gpt-6.1-sol`, `gpt-6-astra`, `gpt-6-sol`, `gpt-6-luna`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.6-cyber`, `gpt-5.5`, `gpt-5.5-pro`, `gpt-5.4`, `gpt-5.4-pro`, `gpt-5.4-mini`, `gpt-5.2`, `gpt-5.2-pro`, `gpt-5`, `gpt-5-mini`, `o3` | | Anthropic | `claude-fable-5-2`, `claude-fable-5-1`, `claude-fable-5`, `claude-opus-5-5`, `claude-opus-5`, `claude-opus-4-8`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-opus-4-5`, `claude-sonnet-5-5`, `claude-sonnet-5`, `claude-sonnet-4-6`, `claude-haiku-5-5`, `claude-haiku-4-5` | | Google Gemini | `gemini-3.8-flash`, `gemini-4-argon`, `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite`, `gemini-3.1-flash-lite`, `gemini-3.1-pro-preview`, `gemini-3-flash-preview`, `gemini-2.5-flash`, `gemini-2.5-pro` | | DeepSeek | `deepseek-v4-pro`, `deepseek-flash`, `deepseek-v4-flash` | | xAI | `grok-4.7`, `grok-4.6`, `grok-4.5`, `grok-4.3`, `grok-4.20`, `grok-4.20-0309-non-reasoning`, `grok-4`, `grok-build-0.1` | | Moonshot (Kimi) | `kimi-k3`, `kimi-k2.7-code`, `kimi-k2.7-code-highspeed`, `kimi-k2.6`, `kimi-k2.5`, `kimi-k2-thinking` | | Z.AI (GLM) | `glm-5.2`, `glm-5.3`, `glm-5.3-flash`, `glm-5.3-flashx`, `glm-5.1`, `glm-5`, `glm-5-turbo`, `glm-4.7`, `glm-4.7-flash`, `glm-4.7-flashx`, `glm-4.6`, `glm-4.5`, `glm-4.5-air`, `glm-4.5-airx`, `glm-4.5-flash` | | TypeSafe AI | `jev-1.13.0`, `jev-1.14.0` | TypeSafe AI runs detectors only; it does not appear in the chat or RCA model pickers. **Note:** Azure OpenAI, AWS Bedrock, and OpenRouter accept free-text model IDs instead of using a curated catalog. ## Setup 1. Go to Workspace Settings -> Model Providers in the dashboard 2. Select a provider 3. Enter your API key 4. Optionally set a custom base URL for self-hosted or proxied models 5. Save — keys are encrypted at rest and scoped to your workspace Once configured, the model selector in the AI chat interface will show both system models and your BYOK models. ![Settings page with Model Providers selected, showing Add Provider and Configured Providers with keys masked](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/model_providers_v1.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=f7f241d54c04396a72582cb19cc1afd2) Model Providers: Add Provider and configured providers with masked keys # GitHub Integration Source: https://traceroot.ai/docs/ai-agent/github-integration Connect your GitHub repositories to enable source code access and automated PRs GitHub integration lets the traceroot Agent clone your repo in a sandbox, read the source code behind failing spans, and create fix PRs — all from within traceroot. ## Setup 1. Go to Workspace Settings -> Integrations in the traceroot dashboard and Connect GitHub App. 2. Grant access to the repositories you want to connect ![Workspace Settings Integrations page with GitHub and Slack connected](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/integrations_v1.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=c996e4e8c6679c49052a560fa52b825f) Workspace Integrations: Connected Services with GitHub and Slack ## Permissions | Permission | Access | Why | | - | - | - | | Metadata | Read-only | List repositories, collaborators, and access repository metadata (mandatory) | | Contents | Read and write | Clone and read source code for debugging; create fix PRs | | Pull requests | Read and write | Create fix PRs from root cause analysis; read PR context | | Issues | Read and write | Create and manage issues; read issue context for debugging | | Commit statuses | Read-only | Read commit statuses for trace correlation | | Discussions | Read-only | Read discussions and related context | ## Requirements For source code correlation to work: * The traceroot GitHub App must have access to the repository * Traces must include git context typically auto-detected by the SDK > [!NOTE] > > Git context is captured automatically for every span when you use the `@observe` decorator. See [Git Context](/docs/tracing/git-context). # Investigate traces with the traceroot Agent Source: https://traceroot.ai/docs/ai-agent/overview Ask why a run failed and have the agent open a pull request with a fix. The traceroot Agent is built into traceroot and serves as both an **investigative trace debugger** and an **interactive project assistant**. The agent can analyze execution traces, clone production code into sandboxes to locate root causes, and open fix pull requests on GitHub. Beyond investigation, the agent can also modify your project by creating, updating, and deleting monitoring resources—in an attended chat session every change is proposed for your approval before it runs, and a delete never runs without one. ![traceroot Home page assistant with three starter prompts and the chat input](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/agent_home_v1.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=203952b1c9b9fa481b7ab89fdcaa46e6) Home: starter prompts and the chat input ## The Home Page & Persistent Chat The first item in the traceroot project sidebar is **Home**, a full-page AI assistant with starter prompts for common tasks: * *"Investigate the most recent errored trace"* * *"Summarize today's sessions"* * *"What did detector findings flag this week?"* ### Conversation Persistence Chat state lives in the app-wide assistant provider, scoped to the project you are in: * **Navigates with you**: A conversation started on the Home page survives navigating across different pages (such as Traces, Dashboards, or Detectors) and remains accessible in the right-hand assistant drawer. * **Stored sessions**: Conversations are saved as project sessions, allowing you to review past debugging trajectories and findings at any time. * **Turn-level trace links**: Assistant responses include a **View trace** button when their trace is available or still uploading; failed trace exports are marked unavailable. *** ## Investigative Capabilities When debugging issues in your AI applications, the agent provides four core investigation capabilities: * **Trace analysis** — Ask natural language questions about your traces, error trends, latencies, and token costs without writing manual ClickHouse queries. * **Root cause analysis (RCA)** — The agent clones your production repository into an isolated sandbox, reads the exact source file and line where an error occurred, and correlates git commit history with trace failures. * **Automated fix PRs** — After diagnosing a failure, the agent can implement a code fix on a new branch and open a GitHub pull request for human review. The agent never pushes directly to your default branch. * **Parallel runs** — Launch multiple debugging sessions concurrently to investigate separate traces and failure patterns simultaneously. *** ## Changing Project Resources (Write Actions) In addition to querying telemetry, the agent binds **13 write tools** capable of modifying project resources directly through conversation: | Resource Type | Available Agent Actions | Write Tools | | :- | :- | :- | | **Detectors** | Create, configure, or delete LLM-as-a-judge detectors | `create_detector`, `update_detector`, `delete_detector` | | **Dashboards** | Create, rename, or delete project dashboards | `create_dashboard`, `update_dashboard`, `delete_dashboard` | | **Widgets** | Add, modify, or remove query charts and trace feeds | `create_widget`, `update_widget`, `delete_widget` | | **Alerts** | Create, reconfigure, delete, or toggle threshold alerts | `create_alert`, `update_alert`, `delete_alert`, `set_alert_status` | ### Scope Boundary: No Workspace or Project Writes The agent operates strictly within the project tenancy injected by the active chat session: * The agent **cannot create, update, or delete workspaces**. * The agent **cannot create, update, or delete projects**. Administrative changes to workspaces and projects are intentionally restricted to the CLI, Public REST API, and workspace settings. *** ## Safety: Interactive Approvals and Timeouts In an attended chat session, every write the agent proposes pauses for your decision before it runs: ### Confirmation Cards Whenever the agent decides to invoke a write tool, execution pauses and presents an interactive **confirmation card** in the chat: * **Creates and Updates**: You can choose to **Approve** (execute the change), **Skip** (cancel the change), or **Revise** (provide feedback or changes in natural language so the agent re-proposes the action with adjustments). * **Deletes**: To protect against accidental deletions, destructive actions offer only **Approve** or **Skip**. A skipped deletion is never re-proposed from revisions. If you skip or decline an action, the agent acknowledges the skip and continues the conversation without retrying the tool call. > [!NOTE] > > For creates and updates, the confirmation card is a review gate rather than a security control: a session with no user attached to it, such as a system-initiated run, executes them without showing a card. Deletes fail closed instead — with nobody there to approve one, the call is blocked rather than executed. ### 10-Minute Timeout Approval cards will not wait indefinitely. If an approval proposal is left unanswered for **10 minutes**, traceroot automatically skips that pending action. No change from the unanswered proposal is applied; earlier approved actions in the same turn remain in effect. *** ## Audit Logging Writes executed by the agent are recorded on your workspace's append-only audit trail: * **Transport distinction**: Audit rows created by the agent are marked with `transport: "agent"` (differentiating them from `"public-api"` or manual UI writes). * **Session correlation**: Each entry stores the `agentSessionId`, linking the modified resource to the agent conversation that authorized the change. * **Persistence**: Audit rows survive resource deletion, so administrators keep a historical record of who approved what change, when, and through which conversation. The audit row is written after the resource change itself commits, and a failed audit write is logged rather than rolled back — so the trail is append-only but not a guarantee that every write has a row. *** ## Where to start > [card] [GitHub Integration](/docs/ai-agent/github-integration) > > Connect your repositories to enable source code access and PR creation. > [card] [BYOK](/docs/ai-agent/byok) > > Use your own API keys to choose which model powers the agent. > [card] [Trace Analysis](/docs/ai-agent/trace-analysis) > > Ask the agent questions about your traces in natural language. > [card] [Root Cause Analysis](/docs/ai-agent/root-cause-analysis) > > Learn how the agent identifies the root cause of failures. # Automated PR Creation Source: https://traceroot.ai/docs/ai-agent/pr-creation The traceroot Agent writes fixes and opens pull requests on GitHub After identifying a root cause, the traceroot Agent writes a fix and opens a pull request on GitHub — linked back to the trace that surfaced the issue. Requires setting up [GitHub Integration](/docs/ai-agent/github-integration). ## How It Works 1. **Root cause analysis** — the agent identifies the bug (see [Root Cause Analysis](/docs/ai-agent/root-cause-analysis)) 2. **Write the fix** — modifies the source code in the sandbox 3. **Open the PR** — pushes a new branch and creates a PR on GitHub for you to review The agent never pushes directly to main. # Root Cause Analysis Source: https://traceroot.ai/docs/ai-agent/root-cause-analysis How the traceroot Agent identifies the root cause of failures in your traces When a trace contains errors, the traceroot Agent finds the root cause by looking at trace data, prod source code and GitHub activities. Requires setting up [GitHub Integration](/docs/ai-agent/github-integration). ## How It Works 1. **Examine the trace** — reviews error spans, call hierarchy, input/output data, and timing 2. **Clone the repo** — clones your repository at the relevant commit into an isolated sandbox 3. **Read the prod source code** — reads the exact file and line captured in the span attributes 4. **Identify the root cause** — correlates trace data with source code and explains what went wrong ## Sandbox Isolation The agent runs in a fully isolated sandbox — it can read trace data and clone your source code, but cannot access your production databases, APIs, or infrastructure. # Trace Analysis Source: https://traceroot.ai/docs/ai-agent/trace-analysis Ask the traceroot Agent questions about your traces in natural language Ask the traceroot Agent questions about your traces directly from the chat interface in the traceroot dashboard. The agent can search, filter, and aggregate across your full trace history. ![traceroot Agent summarizing the traces from the last 7 days](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/trace_summary_v2.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=4408252a9026ca4063211a8076132144) traceroot Agent: a question about recent traces, the queries it ran, and the summary ## Example Queries **Errors** * "Summarize the errors in the last hour" * "What's the most common failure mode for this agent?" * "Show me all traces where the search tool failed" **Performance** * "Which agent has the highest latency?" * "What's the average cost per trace this week?" * "Compare latency between gpt-4o and gpt-4o-mini calls" **Data exploration** * "How many traces did user-123 generate today?" * "What's the total token spend this month?" * "Show me a breakdown of model usage across all traces" ## Tips * Be specific about time ranges — "last hour" is better than "recent" * Mention specific services or agents to narrow the search * Ask follow-up questions — the agent maintains context within a conversation # Alerts for agent latency, cost, and errors Source: https://traceroot.ai/docs/alerts/overview Set a threshold and get a Slack message when a metric crosses it or returns to normal. Alerts watch your agent's latency, cost, token use and errors, and post to Slack when one crosses a threshold you set. You are notified about a problem when it starts, without having to watch a dashboard. Alerts are different from [detectors](/docs/detectors/introduction): a detector judges one trace with a prompt; an alert watches a number over time. Try Alerts and Slack delivery now on any plan, including Free and self-hosted. A project can hold up to 100 alerts. ![Alerts list with four rules in the Alert, OK and No Data states](https://mintcdn.com/tracerootai/iLgDdz0HuQBcV2gH/images/alerts_list_v2.png?fit=max\&auto=format\&n=iLgDdz0HuQBcV2gH\&q=85\&s=d99770193cb132b97747611903470da1) Alerts list: each rule's state, window and last evaluation ## How alerts work A rule reads as one sentence: *aggregation* of *measure*, over spans matching the *filters*, in the last *window*, compared to a *threshold*. For example, `p95` of **Latency** over LLM spans in the last `10m`, `>` `2000` ms. Each evaluation reads **OK**, **Alert** or **No Data**, and only a change notifies: * Entering **Alert** sends a message, and recovering to **OK** sends another. * A rule that stays in **Alert** is quiet unless **Renotify** is on. * To be told when traffic stops, alert on **Count** `<` `1`. Count rules read `0` on an empty window, never **No Data**. * Other rules read **No Data** on an empty window and stay quiet, by default. **Treat as zero** compares `0` to the threshold instead, and **Notify when data stops** sends a message for the gap. ## How to create an alert You need incoming traces, the **Member** role or higher, and Slack connected to the workspace (see [Slack notifications](#slack-notifications)). Open your project in [traceroot Cloud](https://app.traceroot.ai), select **Alerts** from the left nav, then click **New Alert**. 1. **Scope**: click **Add filter** to limit which spans are measured, such as `Span kind is LLM`. Leave it empty to measure every span. 2. **Condition**: pick the aggregation and measure, the operator and threshold, and the window. The threshold is in the measure's unit, so "p95 latency over 2 seconds" is `2000`. A rule has one threshold; for a warning and a critical level, create two alerts. 3. **Notify**: set **When a window has no data** and **Renotify**, and enter a **Name**. The defaults notify only on a breach and a recovery. **Integration** shows the Slack channel the alert posts to. 4. Click **Create Alert**. The rule is active immediately, and its first result appears within a minute. The **Live preview** charts the rule as you edit, with the threshold drawn across it. If it shows **No preview available for this metric yet**, the measure, aggregation and filters cannot be combined, and **Create Alert** stays disabled. ## Create alerts with AI Select **Home** from the left nav to open the traceroot Agent, then describe the alert you want, or ask it to suggest one from your traces. For example: * "Alert me when p99 latency of LLM spans goes over 2 seconds" * "Suggest a useful alert for this project based on my recent traces" The agent drafts the rule with its preview chart. Nothing is saved until you click **Create alert** under the chat; reply instead to revise the draft. ![traceroot Agent checking existing alerts and recent traffic, then creating a span error alert](https://mintcdn.com/tracerootai/iLgDdz0HuQBcV2gH/images/alerts_agent_v2.png?fit=max\&auto=format\&n=iLgDdz0HuQBcV2gH\&q=85\&s=cb2bf33229eae4e410f56ef82481bd75) traceroot Agent: an error alert suggested from recent traces, then approved in chat AI coding agents can also create alerts with the [traceroot CLI](/docs/cli/get-started) (`traceroot alerts create`) or the public API under `/api/v1/public/alerts`. Writes need a signed-in user; a project API key can only read. ## Alert states Each row in the Alerts list shows a badge. When a badge has a reason, such as why a notification was not delivered, click it to read it. * **OK**: the latest value is within the threshold. * **Alert**: the latest value breaches the threshold. * **No Data**: the latest window was empty, or the rule has not run yet. * **Failing**: the last run failed. It retries on its normal schedule. * **Parked**: traceroot stopped the rule because its settings cannot be evaluated. Fix the setting the badge names and save, or resume the rule. * **Paused**: you stopped the rule. Nothing is evaluated or sent. ## Manage alerts Use the **Actions** menu on a row: * **Pause** stops evaluation and notifications. **Resume** starts the rule fresh. * **Edit** opens the same form. Saving a change to the rule starts it fresh; a rename or a renotify change does not. * **Delete** removes the rule permanently. Type the alert's name to confirm. Starting fresh closes an open alert and posts an `[OK]` to Slack that says who edited or resumed the rule; see [Starting fresh](/docs/alerts/rule-reference#starting-fresh). The edit form shows a reminder when the rule has an alert open. Deleting a rule with an alert open closes it the same way: the `[OK]` names who deleted the rule when their account has a display name, and has no link, since the rule is gone. The delete dialog shows a reminder too. ## Slack notifications Slack is connected once per traceroot workspace, and every project in it posts alerts to that channel. Detector notifications use the same connection. 1. As a workspace admin, open **Settings → Integrations**. 2. Click **Connect** and complete the authorization in Slack. 3. Pick the channel traceroot should post to. Each message names the state and the rule, such as `[ALERT] p95 latency`, then states the value, threshold, window and filters in one sentence. **View alert** opens the rule, and a breach also links **View traces** for the evaluated window. The `[OK]` that closes an alert after an edit or a resume is not a recovery, so it states who made the change in place of a value. ![Slack message for a p95 latency breach with View alert and View traces links](https://mintcdn.com/tracerootai/iLgDdz0HuQBcV2gH/images/alerts_slack_v2.png?fit=max\&auto=format\&n=iLgDdz0HuQBcV2gH\&q=85\&s=3deea2a97f1f195d3cb79996875ac31d) Slack message: a breach with its value, threshold, window and links ## Next steps > [card] [Rule Reference](/docs/alerts/rule-reference) > > Every measure, unit, aggregation, filter and state. > [card] [Detector Notifications](/docs/detectors/introduction#notifications) > > Email and Slack digests for new and reopened detector signals. # Rule Reference Source: https://traceroot.ai/docs/alerts/rule-reference Complete reference for alert settings and states This reference covers each setting of an alert rule, the states a rule can be in, and the limits that apply. For a step-by-step guide to your first alert, see the [Introduction](/docs/alerts/overview). ## Measures Alerts measure spans. Each measure has a unit, and the threshold you enter is in that unit. | Measure | ID | Unit | What it reads | | - | - | - | - | | Count | `count` | Spans | Number of spans in the window. | | Trace ID | `trace_id` | Traces | The trace each span belongs to. Aggregate with `uniq` to count distinct traces. | | Latency | `latency` | Milliseconds | Elapsed time of one span, from its start to its end. | | Cost | `cost` | USD | Cost recorded on one span. | | Input tokens | `input_tokens` | Tokens | Input tokens recorded on one span. | | Output tokens | `output_tokens` | Tokens | Output tokens recorded on one span. | | Total tokens | `total_tokens` | Tokens | Input plus output tokens recorded on one span. | | Total tokens per second | `total_tokens_per_second` | Tokens per second | A span's total tokens over its duration. A span with no measured duration has no value. | | Unique user ids | `unique_user_ids` | Users | The user a trace belongs to. Aggregate with `uniq` to count distinct users. | | Unique session ids | `unique_session_ids` | Sessions | The session a trace belongs to. Aggregate with `uniq` to count distinct sessions. | ## Aggregations Each measure accepts these aggregations: | Measures | Aggregations | | - | - | | Count | `count` only | | Trace ID, Unique user ids, Unique session ids | `uniq` only | | Latency, Cost, Input tokens, Output tokens, Total tokens, Total tokens per second | `sum`, `avg`, `min`, `max`, `p50`, `p75`, `p90`, `p95`, `p99`, `uniq` | The form offers only valid pairs; the API rejects the rest. `uniq` counts distinct values, so its result is a count rather than a value in the measure's unit. ## Operator and threshold The aggregated value is compared to the threshold with one of six operators. The form shows the symbol, and Slack messages spell it out. | Operator | In the form | In Slack | | - | - | - | | `>` | > | above | | `>=` | ≥ | at or above | | `<` | \< | below | | `<=` | ≤ | at or below | | `=` | = | equal to | | `!=` | ≠ | not equal to | A rule has a single threshold, between `-1e34` and `1e34`. For two levels, such as a warning and a critical level, create two alerts. ## Window The window is how far back each evaluation looks: **1m**, **5m**, **10m**, **30m**, **1h** or **2h**. The default is **10m**. The window sets how much recent data each check reads, not how often you are notified. A rule with a `1m` window is checked every minute; every other rule is checked every 5 minutes. Each window ends 30 seconds ago, so spans that arrive late still count. ## Filters Filters limit which spans are measured. A span must match every filter on the rule. The form ignores half-filled rows. | Field | Matches | Operators | | - | - | - | | `model_name` | The model a span called | `=`, `contains` | | `environment` | The environment the span was recorded in | `=`, `contains` | | `status` | The span's status, `OK` or `ERROR` | `=`, `contains` | | `span_kind` | The span's kind, such as `LLM`, `AGENT` or `TOOL` | `=`, `contains` | | `name` | The span's name | `=`, `contains` | | `is_root` | Whether the span is the root of its trace (`true` or `false`) | `=` | | `metadata` | A metadata value on the span. Requires a key. | `=`, `contains` | **Unique user ids** and **Unique session ids** work only on rules without filters. ## When a window has no data `count` and `uniq` read `0` on an empty window, so this setting does not apply to them. To be notified when traffic stops, alert on `count` `<` `1`. Every other aggregation, `sum` included, is empty when the window has no spans that record the measure, such as a cost sum over spans without a cost. **When a window has no data** decides what happens: | Mode | In the form | Behavior | | - | - | - | | `HOLD` (default) | Show no data, don't notify | The rule reads **No Data**. Nothing is sent and nothing clears. An alert that was already open stays open until data returns. | | `ZERO` | Treat as zero | An empty window counts as `0` and is compared to the threshold as usual. | | `NOTIFY` | Notify when data stops | The gap itself sends a notification, and the return of data sends another. With renotify on, the gap message repeats on the same interval. | Under `NOTIFY`, the message goes out once the gap has lasted as long as the rule's window, or 10 minutes for longer windows. ## Renotify Renotify controls whether a rule that stays in breach repeats its message. * **Off (alert only on transitions)**, the default: the rule notifies only when it enters **Alert** and when it recovers. * **Re-alert at a regular interval**: the breach message repeats every *N* minutes while the rule stays in **Alert**. The interval defaults to 60 minutes. ## Severities The severity is the result of the latest evaluation. The list badge shows it, or the status when the rule is not active. | Severity | Shown as | Meaning | | - | - | - | | `UNKNOWN` | No Data | The rule has not been evaluated yet, or was just edited or resumed. | | `OK` | OK | The value is within the threshold. | | `ALERT` | Alert | The value breaches the threshold. | | `NO_DATA` | No Data | The window had nothing to measure. | A rule whose last run failed shows **Failing** in the list, with the error. It retries on its normal schedule: next minute for a `1m` window, within 5 minutes for wider ones. ## Statuses The status says whether the rule is being evaluated at all. | Status | Meaning | | - | - | | `ACTIVE` | The rule is evaluated on schedule. | | `PAUSED` | You stopped the rule. It is not evaluated and sends nothing until resumed. | | `PARKED` | traceroot stopped the rule because its saved settings cannot be evaluated. | Only traceroot sets `PARKED`, and the badge states the reason. To start a parked rule again, change the setting named in the reason and save, or resume it. A save that changes nothing, or only the name, leaves it parked. A parked rule cannot be paused. ## Starting fresh Some actions start a rule fresh: its severity returns to **No Data** until the next evaluation, and an open alert closes. If the metric still breaches, the next evaluation announces a new alert. Closing an open alert queues an `[OK]` message for Slack, normally within about a minute. It is not a recovery: it carries no measured value, and names who edited or resumed the rule. When the metric still breaches, the new `[ALERT]` follows the `[OK]`. A rule edited while paused still sends it, because the alert it closes was already announced. If Slack keeps refusing the message for about 30 minutes it is queued again, up to three more times; if no channel is set, the Slack connection is incomplete, or Slack rejected it, it is not sent again. Deleting a rule with an alert open posts the same `[OK]` right away, naming who deleted the rule when their account has a display name. It is sent once: if no channel is set or Slack does not accept it, it is not sent again, and the delete still goes through. | Action | Starts fresh | | - | - | | Resume | Always | | Edit filters, measure, aggregation, operator, threshold, window or no-data mode | Yes | | Edit renotify only | No, but it restarts a parked rule | | Rename, or save with no change | No | | Pause | No; evaluation stops | ## Limits * **100 alerts per project**, on every plan. Paused and parked alerts count toward the limit. The list shows how many are used once fewer than 10 are left, and **New Alert** is disabled at the limit. * **Names** are required, up to 200 characters, and not unique: creating the same rule twice makes two rules. * **50 filters** per rule. * **Renotify** interval from 1 to 10,080 minutes. ## Next steps > [card] [Introduction](/docs/alerts/overview) > > Create your first alert and connect Slack. > [card] [traceroot Agent](/docs/ai-agent/overview) > > The in-app agent that can draft alerts for you. # Create a project Source: https://traceroot.ai/docs/api-reference/account/create-a-project /api-reference/openapi.json post /api/v1/public/projects Create a project in a traceroot workspace you can write to. The call is idempotent on the project name within the workspace. Use a user credential with the `MEMBER` role or higher in the workspace. # Create a workspace Source: https://traceroot.ai/docs/api-reference/account/create-a-workspace /api-reference/openapi.json post /api/v1/public/workspaces Create a traceroot workspace that you administer. Creating one with a name you already administer returns the existing workspace. # Delete a project Source: https://traceroot.ai/docs/api-reference/account/delete-a-project /api-reference/openapi.json delete /api/v1/public/projects/{project_id} Delete a traceroot project. It disappears from lists and reads and its access keys stop working, while its data stays for the retention window. Use a user credential with the `ADMIN` role in the workspace. # Delete a workspace Source: https://traceroot.ai/docs/api-reference/account/delete-a-workspace /api-reference/openapi.json delete /api/v1/public/workspaces/{workspace_id} Permanently delete a traceroot workspace and everything in it, including projects, access keys, memberships, and invites. Requires the ADMIN role. Use a user credential with the `ADMIN` role in the workspace. Send the workspace's current name to confirm deletion. You cannot delete your only workspace. # Edit a project Source: https://traceroot.ai/docs/api-reference/account/edit-a-project /api-reference/openapi.json patch /api/v1/public/projects/{project_id} Edit a traceroot project's name or trace retention period. Setting the retention to null returns it to the default for your plan. Use a user credential with the `ADMIN` role in the workspace. # List Projects Source: https://traceroot.ai/docs/api-reference/account/list-projects /api-reference/openapi.json get /api/v1/public/projects List the traceroot projects the signed-in user can access across all workspaces, to find the project id that project-scoped requests need. Use a user credential. Each project includes its owning workspace. # List Workspaces Source: https://traceroot.ai/docs/api-reference/account/list-workspaces /api-reference/openapi.json get /api/v1/public/workspaces List the traceroot workspaces the signed-in user belongs to. Use it with the project list to find the project a later request should target. Use a user credential. This request does not take a project ID. # Rename a workspace Source: https://traceroot.ai/docs/api-reference/account/rename-a-workspace /api-reference/openapi.json patch /api/v1/public/workspaces/{workspace_id} Rename a traceroot workspace that you administer. Requires the ADMIN role, and the new name cannot match another workspace you administer. Use a user credential with the `ADMIN` role in the workspace. A name that matches another workspace you administer returns 409. A workspace outside your memberships returns 403. # Create an alert Source: https://traceroot.ai/docs/api-reference/alerts/create-an-alert /api-reference/openapi.json post /api/v1/public/alerts Create a threshold alert in a traceroot project: a span measure aggregated over a time window and compared against a threshold you set. Use a user credential with the `MEMBER` role or higher in the workspace. Each call creates a new alert, even when its name matches an existing one. # Delete an alert Source: https://traceroot.ai/docs/api-reference/alerts/delete-an-alert /api-reference/openapi.json delete /api/v1/public/alerts/{alert_id} Permanently delete a threshold alert from a traceroot project. An open page is discarded rather than resolved, and the response reports it. Use a user credential with the `MEMBER` role or higher in the workspace. # Edit an alert rule Source: https://traceroot.ai/docs/api-reference/alerts/edit-an-alert-rule /api-reference/openapi.json patch /api/v1/public/alerts/{alert_id} Update a threshold alert in a TraceRoot project. Changes to evaluated fields reset its state, while changing only the alert name preserves it. Use a user credential with the `MEMBER` role or higher in the workspace. Changing an evaluated field resets evaluation, clears an open page, and re-arms a parked alert. Renaming the alert alone leaves its evaluation state unchanged. # Get Alert Source: https://traceroot.ai/docs/api-reference/alerts/get-alert /api-reference/openapi.json get /api/v1/public/alerts/{alert_id} Fetch one traceroot alert by id with its full rule: view, measure, aggregation, filters, window, threshold, and current evaluation state. # List Alerts Source: https://traceroot.ai/docs/api-reference/alerts/list-alerts /api-reference/openapi.json get /api/v1/public/alerts List the threshold alerts in a traceroot project with each alert's rule summary, status, and current severity, plus the project's alert capacity. Alerts are ordered oldest first. # Pause or resume an alert Source: https://traceroot.ai/docs/api-reference/alerts/pause-or-resume-an-alert /api-reference/openapi.json patch /api/v1/public/alerts/{alert_id}/status Pause or resume a traceroot alert without changing its rule. Pausing keeps the current severity, and resuming restarts evaluation from a clean state. Use a user credential with the `MEMBER` role or higher in the workspace. Setting the current status again returns 200 without changing the alert. # Authentication Source: https://traceroot.ai/docs/api-reference/authentication Which credential each API route accepts, and how to get one Every request to the traceroot API sends a credential as a bearer token: ```bash theme={null} curl "https://app.traceroot.ai/api/v1/public/traces?limit=10" \ --header "Authorization: Bearer $TRACEROOT_API_KEY" ``` The API accepts three kinds of credential. Which ones a route takes depends on what the route does, so check the table under [Which routes accept which credential](#which-routes-accept-which-credential) before you pick one. ## Credentials | Credential | Looks like | Scope | Lifetime | | - | - | - | - | | Project API key | `tr-` followed by a UUID | One project | Until deleted, or until its optional expiry | | Session token | An opaque string | You, across every project you can access | Your login session | | CLI access token | A JWT (`header.payload.signature`) | You, across every project you can access | 10 minutes | ### Project API key Create a key in your project's settings in [traceroot Cloud](https://app.traceroot.ai) or your [self-hosted](/docs/developer/self-hosting) dashboard. The full key is shown once, when you create it, so store it somewhere safe. A key belongs to one project, so requests made with it need no `project_id`. If you send one anyway, it must match the key's project, or the request fails with `400`. ### Session token and CLI access token Both identify you rather than a project. Sign in with the [CLI](/docs/cli/get-started): ```bash theme={null} traceroot login ``` This runs a browser login and stores a session token in `~/.config/traceroot/credentials.json`. You can send that session token as the bearer directly. The CLI itself sends a short-lived access token instead. To mint one, exchange the session token at the web app: ```bash theme={null} curl --request POST "https://app.traceroot.ai/api/cli/token" \ --header "Authorization: Bearer $TRACEROOT_TOKEN" ``` The response holds `accessToken`, valid for 10 minutes. Mint a new one when it expires. ## Which routes accept which credential | Routes | API key | Session token or CLI access token | | - | - | - | | Reads: traces, sessions, detectors and findings, dashboards, widgets (including `POST /widgets/query`), alerts, SQL (including `POST /sql`), datasets, evaluations and evaluation runs | Yes | Yes, with `project_id` | | Trace ingest (`POST /traces`) and evaluation-run writes (`POST /evaluation-runs`, `.../results`, `.../complete`) | Yes | No | | Dataset authoring (`POST /datasets`, `PATCH /datasets/{dataset_id}`, `POST /datasets/{dataset_id}/versions`), used by the SDKs and not listed in this reference | Yes | No | | `GET /whoami` | Yes | No | | Account reads: `GET /projects`, `GET /workspaces` | No (`403`) | Yes | | Creating, editing and deleting projects, workspaces, alerts, dashboards, detectors and widgets, and pausing or resuming alerts | No (`403`) | Yes | ### Reads with a user credential need `project_id` Except for the account reads above, a session token or access token is not tied to a project, so a project-scoped read made with one must name the project as a query parameter: ```bash theme={null} curl "https://app.traceroot.ai/api/v1/public/traces?project_id=$PROJECT_ID&limit=10" \ --header "Authorization: Bearer $TRACEROOT_TOKEN" ``` Without it, the request fails with `400` and `project_id query parameter is required for user credentials`. Call `GET /projects` to list the project IDs you can use. ### Writes need a user login Creating, editing and deleting resources acts on your account, so these routes reject API keys with `403` and `this operation requires a user login; API keys are project-scoped`. Send a session token or access token instead. For detector, dashboard, widget and alert writes, send `project_id` in the create or edit body, or in the query on delete. Project updates and deletes take the project from the `{project_id}` path parameter. A write made with an access token is also refused with `401` once you run `traceroot logout`, even before the token expires. ### Ingest is API-key only The SDKs send traces and evaluation results with a project API key. These routes do not accept a user credential. ## Errors | Status | Meaning | | - | - | | `400` | `project_id` is missing on a read with a user credential, or does not match the API key's project. | | `401` | The `Authorization` header is missing or malformed, the credential is invalid, expired or revoked, or a user credential was sent to a route that accepts only a project API key. | | `403` | An API key was sent to a route that needs a user login, or you have no access to the project. | | `503` | traceroot could not verify the credential. Retry the request. | ## Self-hosted On a self-hosted deployment, replace `https://app.traceroot.ai` with your own URL: the backend for `/api/v1/public/*` routes (`http://localhost:8000` by default), and the web app for `/api/cli/token` (`http://localhost:3000` by default). In the playground on each endpoint page, choose the **Self-hosted** server and enter your backend URL. # Create a dashboard Source: https://traceroot.ai/docs/api-reference/dashboards/create-a-dashboard /api-reference/openapi.json post /api/v1/public/dashboards Create a dashboard in a traceroot project. The call is idempotent on the dashboard name, so repeating it does not create a duplicate. Use a user credential with the `MEMBER` role or higher in the workspace. # Delete a dashboard Source: https://traceroot.ai/docs/api-reference/dashboards/delete-a-dashboard /api-reference/openapi.json delete /api/v1/public/dashboards/{dashboard_id} Permanently delete a traceroot dashboard together with all of its widgets. The last remaining dashboard in a project cannot be deleted. Use a user credential with the `MEMBER` role or higher in the workspace. # Edit a dashboard Source: https://traceroot.ai/docs/api-reference/dashboards/edit-a-dashboard /api-reference/openapi.json patch /api/v1/public/dashboards/{dashboard_id} Rename a traceroot dashboard or change its description. Fields you leave out stay unchanged, and a null description clears the current one. Use a user credential with the `MEMBER` role or higher in the workspace. # Get Dashboard Source: https://traceroot.ai/docs/api-reference/dashboards/get-dashboard /api-reference/openapi.json get /api/v1/public/dashboards/{dashboard_id} Fetch one traceroot dashboard by id together with its widgets, including each widget's title, type, query spec, and creation time. # Get Dashboard Data Source: https://traceroot.ai/docs/api-reference/dashboards/get-dashboard-data /api-reference/openapi.json get /api/v1/public/dashboards/{dashboard_id}/data Run the query widgets on a traceroot dashboard for one time window and return their results, so you can read what the dashboard shows. Each widget has its own result. Feeds are skipped, and query widgets beyond the per-request limit return an inline error. # List Dashboards Source: https://traceroot.ai/docs/api-reference/dashboards/list-dashboards /api-reference/openapi.json get /api/v1/public/dashboards List the dashboards in a traceroot project, default dashboard first, with each one's name, description, creator, widget count, and timestamps. # Create a detector Source: https://traceroot.ai/docs/api-reference/detectors/create-a-detector /api-reference/openapi.json post /api/v1/public/detectors Create a detector using a template's built-in instructions or your own prompt. Creating a detector by name is idempotent within a project. Use a user credential with the `MEMBER` role or higher in the workspace. Standard templates use their built-in instructions when you omit `prompt`. For the `blank` template, supply your own prompt. # Delete a detector Source: https://traceroot.ai/docs/api-reference/detectors/delete-a-detector /api-reference/openapi.json delete /api/v1/public/detectors/{detector_id} Permanently delete a detector from a traceroot project. Findings it already produced stay readable, and a repeated delete returns 404. Use a user credential with the `MEMBER` role or higher in the workspace. # Edit a detector Source: https://traceroot.ai/docs/api-reference/detectors/edit-a-detector /api-reference/openapi.json patch /api/v1/public/detectors/{detector_id} Edit a traceroot detector's name, prompt, enabled state, sample rate, or model settings. Only the fields you send change; the template is fixed. Use a user credential with the `MEMBER` role or higher in the workspace. # Get Detector Source: https://traceroot.ai/docs/api-reference/detectors/get-detector /api-reference/openapi.json get /api/v1/public/detectors/{detector_id} Fetch one detector's full configuration by id: prompt, output schema, sample rate, root cause analysis settings, and trigger conditions. # Get Finding Source: https://traceroot.ai/docs/api-reference/detectors/get-finding /api-reference/openapi.json get /api/v1/public/detectors/findings/{finding_id} Fetch a single detector finding by id from a traceroot project, including the full analysis detail the detector produced for the trace. # Get Finding By Trace Source: https://traceroot.ai/docs/api-reference/detectors/get-finding-by-trace /api-reference/openapi.json get /api/v1/public/detectors/traces/{trace_id}/finding Fetch the detector finding attached to a specific trace in a traceroot project. Each trace has at most one finding, so you get one or none. # List Detectors Source: https://traceroot.ai/docs/api-reference/detectors/list-detectors /api-reference/openapi.json get /api/v1/public/detectors List the detectors in a traceroot project, newest first, with each detector's id, name, template, enabled flag, and creation time. # List Findings Source: https://traceroot.ai/docs/api-reference/detectors/list-findings /api-reference/openapi.json get /api/v1/public/detectors/findings List recent detector findings for a traceroot project, newest first, optionally filtered by detector, trace id, or time range. # Complete/finalize an evaluation run Source: https://traceroot.ai/docs/api-reference/offline-eval/completefinalize-an-evaluation-run /api-reference/openapi.json post /api/v1/public/evaluation-runs/{run_id}/complete Mark an evaluation run as completed or failed in traceroot and report its final completeness counts. The traceroot SDKs call this for you. > [!NOTE] > > The TraceRoot SDKs call this endpoint for you; call it directly only where no SDK fits. It accepts a project API key only. # List a dataset's published versions Source: https://traceroot.ai/docs/api-reference/offline-eval/list-a-datasets-published-versions /api-reference/openapi.json get /api/v1/public/datasets/{dataset_id}/versions List the published versions of an evaluation dataset, newest first, each with its test case count and whether it is the current version. # List the project's evaluation datasets Source: https://traceroot.ai/docs/api-reference/offline-eval/list-the-projects-evaluation-datasets /api-reference/openapi.json get /api/v1/public/datasets List the evaluation datasets in a traceroot project, newest first, with each dataset's current published version and cursor-based paging. # List the project's evaluation runs Source: https://traceroot.ai/docs/api-reference/offline-eval/list-the-projects-evaluation-runs /api-reference/openapi.json get /api/v1/public/evaluation-runs List the evaluation runs in a traceroot project, newest first, optionally filtered by evaluation or status, with cursor-based paging. Use the run detail endpoint to read result counts and scores. # List the project's evaluations Source: https://traceroot.ai/docs/api-reference/offline-eval/list-the-projects-evaluations /api-reference/openapi.json get /api/v1/public/evaluations List the evaluations in a traceroot project, newest first, each with its run count and latest run, using cursor-based paging. # Read an evaluation run's summary Source: https://traceroot.ai/docs/api-reference/offline-eval/read-an-evaluation-runs-summary /api-reference/openapi.json get /api/v1/public/evaluation-runs/{run_id} Read an evaluation run's status and counts. Numeric scores are averaged, boolean scores are pass rates, and scores without a supported aggregate are null. Numeric scores return a mean and Boolean scores a pass rate. Categorical, mixed-type, or unobserved scores return null. This endpoint summarizes a single run. # Read one dataset version and a page of its test cases Source: https://traceroot.ai/docs/api-reference/offline-eval/read-one-dataset-version-and-a-page-of-its-test-cases /api-reference/openapi.json get /api/v1/public/dataset-versions/{version_id} Read one immutable evaluation dataset version and its test cases, whole or one page at a time, with inputs and expected values as JSON. Without `limit`, the response includes all cases. For large versions, pass `limit` and follow `next_cursor` until it is null. # Read one evaluation dataset Source: https://traceroot.ai/docs/api-reference/offline-eval/read-one-evaluation-dataset /api-reference/openapi.json get /api/v1/public/datasets/{dataset_id} Read one evaluation dataset by id, including its name, description, key, and current published version, which is null until one is published. # Register an evaluation run Source: https://traceroot.ai/docs/api-reference/offline-eval/register-an-evaluation-run /api-reference/openapi.json post /api/v1/public/evaluation-runs Register and start an evaluation run in traceroot. The call is idempotent on the client run id within an evaluation, so retries are safe. > [!NOTE] > > The TraceRoot SDKs call this endpoint for you; call it directly only where no SDK fits. It accepts a project API key only. # Upsert one test-case result with scores Source: https://traceroot.ai/docs/api-reference/offline-eval/upsert-one-test-case-result-with-scores /api-reference/openapi.json post /api/v1/public/evaluation-runs/{run_id}/results Create or update one test case result and its scores in an evaluation run. The call is idempotent for each run and test case pair. > [!NOTE] > > The TraceRoot SDKs call this endpoint for you; call it directly only where no SDK fits. It accepts a project API key only. # Get Session Source: https://traceroot.ai/docs/api-reference/sessions/get-session /api-reference/openapi.json get /api/v1/public/sessions/{session_id} Fetch one traceroot session with all of its traces. Traces older than your plan's retention window are left out instead of causing an error. # List Sessions Source: https://traceroot.ai/docs/api-reference/sessions/list-sessions /api-reference/openapi.json get /api/v1/public/sessions List the sessions in a traceroot project, where a session groups the traces that share a session id, with the trace count for each session. # Get Signal Source: https://traceroot.ai/docs/api-reference/signals/get-signal /api-reference/openapi.json get /api/v1/public/signals/{signal_id} Fetch one signal by id from a traceroot project, including when it was resolved. A signal that was merged names the signal its findings moved to. # List Signals Source: https://traceroot.ai/docs/api-reference/signals/list-signals /api-reference/openapi.json get /api/v1/public/signals List the signals in a traceroot project, newest first. Signals group the recurring problems that detectors find; merged signals are left out. # Resolve or reopen a signal Source: https://traceroot.ai/docs/api-reference/signals/resolve-or-reopen-a-signal /api-reference/openapi.json patch /api/v1/public/signals/{signal_id}/status Resolve or reopen a signal. After resolution, the detector can reopen the signal only for a matching finding on a trace that started after resolution. Use a user credential with the `MEMBER` role or higher in the workspace. Resolving requires a reason. For `other`, include a `note`. To guard against a concurrent change, send `expected_status`. You cannot dismiss a signal here. # Get Sql Schema Source: https://traceroot.ai/docs/api-reference/sql/get-sql-schema /api-reference/openapi.json get /api/v1/public/sql/schema List the tables and columns, with their types, that a SQL query against your traceroot project may reference. Read it before writing a query. # Run Sql Source: https://traceroot.ai/docs/api-reference/sql/run-sql /api-reference/openapi.json post /api/v1/public/sql Run one read-only SQL query over the spans and traces of your traceroot project and get the rows back. Results are capped and may be truncated. # Export Trace Source: https://traceroot.ai/docs/api-reference/traces/export-trace /api-reference/openapi.json get /api/v1/public/traces/{trace_id}/export Export a trace, its spans, git context, and a manifest. Span input and output are included by default and can be excluded with the fields parameter. The default projection is `full`. Use `fields` to narrow the export. Exports have their own rate limit. # Get Trace Source: https://traceroot.ai/docs/api-reference/traces/get-trace /api-reference/openapi.json get /api/v1/public/traces/{trace_id} Fetch a single trace with its span tree from a traceroot project. The default view is lightweight; request full fields for span input and output. Use `fields=full` or `fields=io,metadata` to include span inputs and outputs. # Ingest Traces Source: https://traceroot.ai/docs/api-reference/traces/ingest-traces /api-reference/openapi.json post /api/v1/public/traces Send OTLP trace data to traceroot as protobuf, optionally gzip-compressed. A 200 response means the batch was stored and queued for decoding. > [!NOTE] > > The TraceRoot SDKs call this endpoint for you; call it directly only where no SDK fits. It accepts a project API key only. Send the body as OTLP protobuf with `Content-Type: application/x-protobuf`, optionally gzip-compressed with `Content-Encoding: gzip`. The default limits are 16 MiB as sent and 32 MiB after decompression. Larger requests return 413. # List Trace Filter Values Source: https://traceroot.ai/docs/api-reference/traces/list-trace-filter-values /api-reference/openapi.json get /api/v1/public/traces/filter-values/{field} List the distinct values currently present for a categorical trace filter field, such as model name or environment, in a traceroot project. # List Traces Source: https://traceroot.ai/docs/api-reference/traces/list-traces /api-reference/openapi.json get /api/v1/public/traces List recent traces in a traceroot project, newest first. Offline evaluation traces are excluded unless you set include_evaluations to true. # Whoami Source: https://traceroot.ai/docs/api-reference/whoami/whoami /api-reference/openapi.json get /api/v1/public/whoami Check which traceroot project and workspace an API key belongs to. Useful for confirming that a credential is valid and correctly scoped. # Create a dashboard widget Source: https://traceroot.ai/docs/api-reference/widgets/create-a-dashboard-widget /api-reference/openapi.json post /api/v1/public/widgets Add a widget to a traceroot dashboard: a query widget that charts a metric, or a feed of recent traces. Every call creates a new widget. Use a user credential with the `MEMBER` role or higher in the workspace. # Delete a dashboard widget Source: https://traceroot.ai/docs/api-reference/widgets/delete-a-dashboard-widget /api-reference/openapi.json delete /api/v1/public/widgets/{widget_id} Permanently delete one widget from a traceroot dashboard. The widget's slot in the dashboard layout is removed along with it. Use a user credential with the `MEMBER` role or higher in the workspace. # Edit a dashboard widget Source: https://traceroot.ai/docs/api-reference/widgets/edit-a-dashboard-widget /api-reference/openapi.json patch /api/v1/public/widgets/{widget_id} Edit a traceroot dashboard widget's title, spec, or display settings. A new spec replaces the old one entirely, and the widget type is fixed. Use a user credential with the `MEMBER` role or higher in the workspace. # Get Widget Source: https://traceroot.ai/docs/api-reference/widgets/get-widget /api-reference/openapi.json get /api/v1/public/widgets/{widget_id} Fetch one saved widget's definition by id, including its title, type, stored query spec, display settings, and the dashboard it belongs to. # Get Widget Data Source: https://traceroot.ai/docs/api-reference/widgets/get-widget-data /api-reference/openapi.json get /api/v1/public/widgets/{widget_id}/data Run one saved traceroot dashboard widget for a time window and return its results, without having to send the widget's query spec again. # Run Widget Query Source: https://traceroot.ai/docs/api-reference/widgets/run-widget-query /api-reference/openapi.json post /api/v1/public/widgets/query Run a widget query spec for a time window and get its rows back, to answer a metric question without saving a widget to a dashboard. # Get started with the traceroot CLI Source: https://traceroot.ai/docs/cli/get-started Set up traceroot in your repository and work with your project from the terminal The traceroot CLI sets up tracing in your repository with one command, then lets you work with your project from the terminal — traces, detector findings, alerts, dashboards, evaluations, and SQL — and pipe the JSON into whatever comes next. It is built for AI coding agents and power users who prefer the command line. Full source and reference live in the [`traceroot-cli`](https://github.com/traceroot-ai/traceroot-cli) repo. ## Set up a repository Run this from the root of the repository you want to trace: ```bash theme={null} npx -y traceroot-cli@latest setup ``` Setup signs you in, has your coding agent add the traceroot SDK, and waits for your first trace; no install or sign-in needed first. [Tracing](/docs/tracing/get-started) covers each step, self-hosting, and troubleshooting. ### Run without prompts Setup never prompts when it isn't attached to a terminal, when `CI` is set to anything but empty or `false`, or with `--no-input` or `--json`. Pass what it would otherwise ask for: ```bash theme={null} npx -y traceroot-cli@latest setup --no-input --api-key "$TRACEROOT_API_KEY" ``` A project API key also sets the project. Without a key, run `npx -y traceroot-cli@latest login` first, then run setup without `--api-key`. Login prints a URL and a code for a person to confirm in a browser, and waits, so a coding agent has to show them to you. If setup exits 10, it can't choose for you, usually because you have several projects or none: follow the remedy it prints. With several projects, you can pass `--project`. Without prompts, setup notes uncommitted changes instead of asking, and writes the instrumentation task to `.traceroot/prompts/setup-instrument.md` rather than running a coding agent. Pass `--agent claude` or `--agent codex` to run one. After the task is done and the application has run, rerun the same setup command with `--resume`, such as `npx -y traceroot-cli@latest setup --no-input --resume`, to get a link to the first trace. [Coding agents](/docs/getting-started/coding-agents#set-up-tracing) has these steps as a prompt. ## Install Every command also runs through `npx`, as above. To keep the `traceroot` command, install the CLI globally: ```bash theme={null} npm install -g traceroot-cli ``` The examples below use `traceroot`. Without a global install, replace it with `npx -y traceroot-cli@latest`. The CLI needs Node.js 20.3 or later, and `setup` needs 20.12 or later. ## Authenticate The CLI authenticates with either a **browser login** (the default interactive device flow) or a **project API key** (for CI environments and scripts). ### Browser login (default) Run `traceroot login` to sign in through your browser: ```bash theme={null} traceroot login ``` It prints a URL and a code, tries to open your browser, and saves a session to `~/.config/traceroot/credentials.json`. If no browser opens, open the URL yourself. ### API key login (CI & scripts) For CI pipelines, automated scripts, or headless environments, use a project API key (`tr-...`): ```bash theme={null} traceroot login --api-key tr-... ``` This validates the key against the API and saves it to `./.traceroot/config.json`. ```bash .env theme={null} TRACEROOT_API_KEY=tr-... TRACEROOT_HOST_URL=https://app.traceroot.ai ``` The CLI resolves credentials in this order: 1. `--api-key` flag 2. `--env-file ` 3. `TRACEROOT_TOKEN` environment variable 4. Credentials file (`~/.config/traceroot/credentials.json`) 5. `TRACEROOT_API_KEY` environment variable 6. Config file (`./.traceroot/config.json`) 7. Auto-discovered `./.env` — a `TRACEROOT_API_KEY` here selects key mode when nothing above it is set Create API keys in [traceroot Cloud](https://app.traceroot.ai) or your [self-hosted](/docs/developer/self-hosting) dashboard. ### Verify credentials Confirm your active identity: ```bash theme={null} traceroot status ``` * **Browser login**: Displays your authenticated email address, host URL, active project (or a hint to select one), and accessible workspaces. * **API key login**: Displays your workspace, project, host URL, and a masked key hint (e.g. `tr-d0a3-57e3`). ## Scope to a project A browser login identifies you across your workspaces rather than binding to one project, and commands such as `traces` and `evals` are project-scoped, so pick a project first. An API key is already scoped to its project, so a key login can skip this step. List the projects you can access to find your project ID: ```bash theme={null} traceroot projects list ``` Set `TRACEROOT_PROJECT_ID` so later commands need no extra flags, or pass `--project ` to individual commands: ```bash theme={null} export TRACEROOT_PROJECT_ID= ``` ## Inspect and export traces Once authenticated and scoped to a project, list, inspect, and export traces: ```bash theme={null} # List recent traces, newest first (filter by time with --since, --from, or --to) traceroot traces list --limit 10 # Inspect one trace: header, duration, span tree, and web app link traceroot traces get # Fetch full trace details including span inputs, outputs, and metadata as JSON traceroot traces get --fields full --json # Write a full trace bundle to disk to share or feed into another tool traceroot traces export --output ./my-trace ``` ### Trace projections and field groups (`--fields`) By default, `traceroot traces get` prints the trace header, an optional detector finding block, the span tree, and the web app URL. When emitting JSON (`--json`), `traceroot traces get` defaults to the lightweight `skeleton` projection, leaving span `input`, `output`, and `metadata` as `null` to minimize payload size. Pass `--fields` to include specific data groups: * **Field groups**: `core`, `usage`, `io`, `metadata` * **Aliases**: `skeleton` (`core,usage`) and `full` (`core,usage,io,metadata`) So `--fields full`, or `--fields io,metadata`, includes span I/O in the JSON payload. `traceroot traces export` instead defaults to the `full` projection, writing `trace.json`, `spans.json`, `git_context.json`, `manifest.json`, and `finding.json` (if flagged by a detector) into the destination directory. Pass `--fields ` to narrow what is exported. ## Read evaluations The CLI also reads your offline [evaluations](/docs/evals/introduction): the datasets you run against and the runs that scored them. ```bash theme={null} # Which evaluations exist, and how each one's latest run ended traceroot evals list # One evaluation's runs, newest first — the RUN ID column is what you read next traceroot evals runs list --evaluation-id # One run: its counts, per-scorer means, and per-case cost and duration traceroot evals runs get ``` See [From the terminal](/docs/evals/cli) for what each read shows, and for the dataset commands. ## Command reference Add `--json` to any command for machine-readable output, or run `traceroot --help` for the full flag list. The `sql` commands are documented in [Read-only SQL](/docs/cli/sql). | Command | Description | | :- | :- | | `setup` | Connect this repository to traceroot and see your first trace. See [Tracing](/docs/tracing/get-started). | | `setup doctor` | Diagnose a setup that did not complete. Flag: `--service `. | | `login` | Sign in with the browser (device flow), or persist an explicit --api-key. | | `logout` | Revoke the session server-side (best-effort) and remove the local credential. | | `status` | Show the identity your credentials resolve to (email or key hint, workspace, project, and host). | | `doctor` | Check local environment, credentials, configuration, repository, and agent skills. | | `instrument` | Generate an agent prompt to instrument this repo with TraceRoot. | | `skills list` | List available AI agent skills and their capabilities. | | `skills install` | Install skills into your coding agent workspace. | | `traces list` | List recent traces for the caller's project (newest first). | | `traces get ` | Get a single trace for the caller's project. | | `traces export ` | Export the trace bundle (trace + spans + git_context + manifest) for the caller's project. | | `detectors list` | List the detectors in the caller's project (newest first). | | `detectors get ` | Get one detector's full configuration for the key's project. | | `findings list` | List recent detector findings for the caller's project (newest first). | | `findings get ` | Get a single finding by id for the key's project. | | `alerts list` | List the alerts in the caller's project (oldest first) with its capacity. | | `alerts get ` | Get one alert with its full rule for the caller's project. | | `alerts create` | Create a threshold alert in a project the authenticated user can write to. | | `alerts update ` | Edit an alert's rule in a project the authenticated user can write to. | | `alerts status ` | Pause or resume an alert without touching its rule. | | `alerts delete ` | Hard-delete an alert. | | `dashboards list` | List the dashboards in the caller's project (default first). | | `dashboards get ` | Get one dashboard with its widgets for the caller's project. | | `dashboards create` | Create a dashboard in a project the authenticated user can write to. | | `dashboards update ` | Edit a dashboard's name or description. | | `dashboards delete ` | Hard-delete a dashboard together with its widgets. | | `workspaces list` | List the workspaces the authenticated user belongs to. | | `projects list` | List the projects the authenticated user can access, across workspaces. | | `workspaces create` | Create a workspace administered by the authenticated user. | | `workspaces update ` | Rename a workspace the authenticated user administers. | | `workspaces delete ` | Hard-delete a workspace and everything in it. | | `projects create` | Create a project in a workspace the authenticated user can write to. | | `projects update ` | Edit a project's name or trace retention. | | `projects delete ` | Soft-delete a project. | | `detectors create` | Create a detector in a project the authenticated user can write to. | | `detectors update ` | Edit a detector in a project the authenticated user can write to. | | `detectors delete ` | Hard-delete a detector; its existing findings stay readable. | | `widgets get ` | Get one saved widget's definition, with its dashboard, for the caller's project. | | `widgets data ` | Answer one saved widget for a window, with a status instead of an exception. | | `widgets create` | Create a widget on a dashboard the authenticated user can write to. | | `widgets update ` | Edit a widget's title, spec, or display config. | | `widgets delete ` | Hard-delete a widget and its entry in the dashboard's layout. | | `sql` | Run one read-only SQL query scoped to the caller's project. | | `sql schema` | Return the curated tables and columns available to public SQL. | | `datasets list` | List datasets, newest first. | | `datasets get ` | Read one dataset. | | `datasets versions list ` | List versions newest-first, each with its case count and whether it is current. | | `datasets versions get ` | Read an immutable snapshot: the version plus its cases, whole or a page at a time. | | `evals list` | List evaluations, newest first, each with its run count and latest run. | | `evals runs list` | List evaluation runs, newest first, optionally one evaluation's or one status's. | | `evals runs get ` | Read a run's own summary: status, result counts, and per-score and per-metric means over the run's results. | ## Output and exit codes With `--json`, a command writes one JSON document to stdout. If it fails, stdout is empty and stderr gets `{"error":{"code":"...","message":"..."}}`. `doctor` and `setup doctor` instead write their report to stdout and exit 1 if a check fails, and `setup` writes one JSON event per line, ending in a `result` event with `ok` and `data.trace_url`. Without `--json`, setup writes everything to stderr, including the link to your first trace. | Exit code | Meaning | | :- | :- | | 0 | Success. For `setup`, also check for the link: it can exit 0 when it's paused or hasn't seen a trace yet | | 1 | Internal error, a failed `doctor` check, or an HTTP error not covered below, such as 409 or a 5xx | | 2 | Usage error, or input the server rejected (HTTP 400 or 422), including a missing `--project` after a browser login | | 3 | Not authenticated, or not allowed | | 4 | Not found | | 5 | Network error or rate limit, retry | | 10 | `setup`: it can't choose for you, usually because there are several projects or none. Follow the remedy it prints | | 11 | `setup`: unsupported language, service path, or coding agent | | 12 | `setup`: the coding agent failed | | 14 | `setup`: no trace arrived in time | | 15 | `setup`: this deployment doesn't support a step | | 16 | `setup`: it would overwrite something unsafely, or couldn't verify an existing key | | 130 | Interrupted | ## Next steps > [card] [Tracing](/docs/tracing/get-started) > > Set up tracing with one command, or instrument by hand. > [card] [Evaluations from the Terminal](/docs/evals/cli) > > Read datasets, evaluations, and past runs. > [card] [traceroot Cloud](https://app.traceroot.ai) > > Create API keys and open traces in the web app. # Read-only SQL Source: https://traceroot.ai/docs/cli/sql Query your project's spans and traces from the CLI or public API traceroot's SQL gateway runs analytical queries against your project's trace data. It is read-only and project-scoped: the project is resolved during authentication, and SQL text and parameters cannot override it. You can query the gateway with the [traceroot CLI](/docs/cli/get-started) or with the public API. ## Query with the CLI Pass a query as one quoted argument, or use `--file ` to read it from a file: ```bash theme={null} traceroot sql "SELECT count() AS spans FROM spans WHERE span_start_time >= now() - INTERVAL 1 DAY" ``` The default output is a table. Use `--csv` for CSV, `--json` for the complete API response, and `--output ` to write the selected format to a file. ```bash theme={null} traceroot sql \ "SELECT span_id, trace_id, name, duration_ms FROM spans WHERE span_start_time >= now() - INTERVAL 7 DAY" \ --csv \ --max-rows 5000 \ --output spans.csv ``` Bind `{name:Type}` placeholders with one `--param name=value` flag per value. Parameters stay separate from the SQL text. ```bash theme={null} traceroot sql \ "SELECT count() FROM spans WHERE span_start_time >= {since:DateTime64(3)} AND environment = {environment:String}" \ --param since=2026-09-18T00:00:00Z \ --param environment=production ``` Run `traceroot sql schema` to list the available tables, columns, and ClickHouse types. The [CLI repository](https://github.com/traceroot-ai/traceroot-cli#sql-queries) documents installation, authentication, output formats, and the complete flag reference. ## Worked queries Always include a time predicate on `span_start_time` or `trace_start_time`. The gateway uses that range when reading the underlying project-scoped views, so bounded queries scan less data. ### Recent error spans ```sql theme={null} SELECT span_id, trace_id, name, status_message, span_start_time FROM spans WHERE status = 'ERROR' AND span_start_time >= now() - INTERVAL 1 DAY ORDER BY span_start_time DESC LIMIT 100 ``` ### p95 latency by model ```sql theme={null} SELECT model_name, quantile(0.95)(duration_ms) AS p95_ms FROM spans WHERE model_name IS NOT NULL AND span_start_time >= now() - INTERVAL 7 DAY GROUP BY model_name ORDER BY p95_ms DESC ``` ### Cost by model over seven days ```sql theme={null} SELECT model_name, sum(cost) AS total_cost FROM spans WHERE model_name IS NOT NULL AND span_start_time >= now() - INTERVAL 7 DAY GROUP BY model_name ORDER BY total_cost DESC ``` ## Query with the API Send a project API key as a bearer token to `POST /api/v1/public/sql`. A user session token also works with the `project_id` query parameter, which is how browser-login users run `traceroot sql --project ` without a project API key: ```bash theme={null} curl --request POST "$TRACEROOT_HOST_URL/api/v1/public/sql" \ --header "Authorization: Bearer $TRACEROOT_API_KEY" \ --header "Content-Type: application/json" \ --data '{ "query": "SELECT model_name, sum(cost) AS total_cost FROM spans WHERE span_start_time >= {since:DateTime64(3)} GROUP BY model_name ORDER BY total_cost DESC", "parameters": {"since": "2026-09-18T00:00:00Z"}, "max_rows": 1000 }' ``` A successful response contains column names and types, row values in the same order, the returned row count, truncation state, elapsed time, and any read statistics supplied by ClickHouse: ```json theme={null} { "columns": [ {"name": "model_name", "type": "Nullable(String)"}, {"name": "total_cost", "type": "Nullable(Decimal(38, 9))"} ], "rows": [["gpt-5", "0.013500000"]], "row_count": 1, "truncated": false, "elapsed_ms": 12, "statistics": {"rows_read": 842, "bytes_read": 91736} } ``` `statistics` is best-effort and may be empty. When more rows match than the gateway returns, `truncated` is `true`. A request's `max_rows` can lower the server cap, but cannot raise it. Fetch the same schema used by the validator and query rewriter with `GET /api/v1/public/sql/schema`: ```bash theme={null} curl "$TRACEROOT_HOST_URL/api/v1/public/sql/schema" \ --header "Authorization: Bearer $TRACEROOT_API_KEY" ``` ## Public schema Queries can reference only two logical tables: | Table | Contents | Time-filter column | | :- | :- | :- | | `spans` | Span identity, timing, status, model, token, cost, environment, metadata, and Git source fields | `span_start_time` | | `traces` | Trace identity, timing, name, user, session, Git, environment, and metadata fields | `trace_start_time` | Use `traceroot sql schema --json` or `GET /api/v1/public/sql/schema` for the authoritative column list and types. These are curated views over traceroot's internal tables. They deliberately exclude: * span and trace `input` and `output` payloads; * the raw metadata JSON document; `metadata` is exposed instead as a one-level string map, accessed with expressions such as `metadata['user_id']`; * `project_id`, `ch_create_time`, `ch_update_time`, and internal classification fields; * rows that are not customer traffic; and * every trace that belongs to an evaluation run, including its spans. ## SQL contract The gateway accepts one `SELECT` or `UNION`. CTEs, subqueries, joins, grouping, ordering, and the `row_number`, `rank`, and `dense_rank` window functions are supported. Functions are allowlisted; the list includes common aggregation, date-bucketing, string, conversion, conditional, and metadata-map functions. A ClickHouse function outside the allowlist is rejected even if ClickHouse normally supports it. The gateway rejects: * multiple statements, writes, DDL, and control statements; * tables other than `spans` and `traces`, including table functions; * database-qualified names and the internal view names; * `FINAL`, `SETTINGS`, and `FORMAT` clauses; * any reference or alias named `project_id`; * a CTE named `spans` or `traces`; and * parameter names `project_id` or beginning with `scope_`, which are reserved for server-side scoping. Queries are limited to 65,536 characters. A request can bind at most 100 parameters, whose serialized JSON is limited to 16,384 characters. ## Execution and rate limits Default execution limits are: | Limit | Default | | :- | -: | | Execution time | 30 seconds | | ClickHouse result rows | 100,000 | | Result size | 512 MiB | | Memory per query | 4 GiB | | Concurrent queries per project, per REST process | 4 | | Concurrent queries in total, per REST process | 16 | The gateway reserves a row beyond the returned result to detect truncation. If that row exists, it removes the sentinel and returns `truncated: true`. Queries that exceed the time, result-size, or memory limits fail; narrow the selected columns or time range and retry. On self-hosted deployments, operators can change the four ClickHouse caps with `CLICKHOUSE_SQL_MAX_EXECUTION_TIME`, `CLICKHOUSE_SQL_MAX_RESULT_ROWS`, `CLICKHOUSE_SQL_MAX_RESULT_BYTES`, and `CLICKHOUSE_SQL_MAX_MEMORY_USAGE`. Concurrency is not a queue. If either per-process slot limit is full, the API returns HTTP 429 with `Too many queries are running. Retry shortly.` and a one-second `Retry-After` header. traceroot Cloud also applies a per-project SQL request rate: | Plan | Requests per minute | | :- | -: | | Free | 20 | | Starter | 60 | | Pro | 120 | | Enterprise | 120 | SQL rate limiting is disabled on self-hosted deployments where billing is disabled. # Get started with dashboards Source: https://traceroot.ai/docs/dashboards/get-started Organize project metrics and trace feeds into reusable dashboards Dashboards let you explore your project's trace data with widgets. Use them to follow latency, cost, token usage, and errors, or keep a trace feed beside your metrics. ## Create a dashboard 1. Open **Dashboard** in your project's navigation. 2. Click **New dashboard**, give it a name, and optionally add a description. 3. Create the dashboard, then add a widget. Creating and editing dashboards or widgets requires MEMBER or ADMIN access. VIEWERs can read existing dashboards. See [workspace roles](/docs/getting-started/concepts#roles). ## Build a widget The widget builder offers fields and aggregations for the selected data view. Choose the metric you want to inspect, apply filters such as environment or model, and select a breakdown when you want to compare groups. Use the preview to check the result before saving the widget. Choose a time range that contains your application's traffic. A dashboard with no matching data is different from a metric whose value is zero; check the time range and filters if a widget has no result. You can edit saved widgets as your monitoring needs change. Dashboards belong to the current project, so switch projects to view another application's dashboards. ## Next steps * [Send trace data](/docs/tracing/get-started) for your widgets to query. * [Set up alerts](/docs/alerts/overview) when a metric needs a threshold notification. * [Understand workspace access](/docs/getting-started/concepts) before sharing the dashboard with teammates. # Detect failures in agent traces Source: https://traceroot.ai/docs/detectors/introduction Describe the problem you want a model to look for, then inspect the traces it flags. A detector checks your agent's traces for one specific problem, such as a hallucination or a failed task. You hear about the problem without reading every trace. For each trace it checks, a model reads the trace and answers yes or no. A yes is a **finding**. traceroot groups findings of the same problem into a [signal](/docs/signals/introduction), and the traceroot Agent can find the signal's root cause. Detectors work on every plan. Free includes 100 detector runs per billing period. ![Detectors list with each detector's template, model, sampling, judge runs, signals and agent runs](https://mintcdn.com/tracerootai/-OSwxtwDVTPazBuT/images/detectors_list_v1.png?fit=max\&auto=format\&n=-OSwxtwDVTPazBuT\&q=85\&s=d1a5ab683b2a557a4964667a2390d952) Detectors list: template, model, sampling, signals and agent runs per detector ## How detectors work Detection runs in two steps. The second step, root cause analysis, costs more, so it runs only on problems the first step finds. **Step 1, the judge.** About a minute after a trace's last span arrives, each detector checks whether the trace matches its filter and its sampling rate. If it does, the detector's model reads the trace, answers whether the problem is present, and writes a one-sentence summary. Detectors skip traces that come from [evaluations](/docs/evals/introduction). **Step 2, the agent.** The traceroot Agent reads one trace of a signal and writes a root cause analysis with a code location and a recommendation. [Signals](/docs/signals/introduction) explains when it runs. Each step has its own model. You pick the judge model on each detector. You pick the agent model once per project, in **Settings → Detectors → Agent Model**. Either can be a model traceroot provides or a model from your own provider key ([bring your own key](/docs/ai-agent/byok)). ## How to create a detector You need a project that receives traces. See [Tracing](/docs/tracing/get-started) to set one up. Open your project in [traceroot Cloud](https://app.traceroot.ai), select **Detectors** from the left nav, then click **New Detector**. ![Detectors page header with the New Detector button highlighted](https://mintcdn.com/tracerootai/-OSwxtwDVTPazBuT/images/detector_new_button_v1.png?fit=max\&auto=format\&n=-OSwxtwDVTPazBuT\&q=85\&s=89412478bd81663128191d535c9c68f1) Detectors page: New Detector button 1. **Template**: pick **Failure** (selected by default), **Hallucination**, **Logic Error**, **Task Completion**, **Safety** or **Blank**. A template fills in the **Name** and the **Prompt**. **Blank** leaves both empty. 2. **Model**: pick the model that judges each trace. The default is `gpt-6-luna`. 3. **Prompt**: describe what to look for, in plain language. 4. **Filter**: choose which traces the detector checks. Leave it empty to check every trace. 5. **Sampling**: set the share of matching traces to check. It starts at 25%. **100%** checks every matching trace, and **0%** pauses the detector. A detector always makes the same sampling choice for a given trace, so sending that trace again does not change whether it is checked. 6. **Signals**: leave **Generate signals** on to group findings into signals. Set **Root cause analysis** as described [below](#root-cause-analysis). 7. Click **Create Detector**. The detector checks traces that arrive from then on. ## Create detectors with AI Select **Home** from the left nav to open the traceroot Agent, then describe the detector you want, or ask it to suggest one from your project. For example: * "Create a detector that flags traces where a tool call fails" * "Suggest a new detector based on my current detectors" The agent drafts the detector. Nothing is saved until you click **Create detector** under the chat; reply instead to revise the draft. ![traceroot Agent listing and reading the project's detectors, then summarizing their setup](https://mintcdn.com/tracerootai/-OSwxtwDVTPazBuT/images/detector_agent_v1.png?fit=max\&auto=format\&n=-OSwxtwDVTPazBuT\&q=85\&s=d656b328cfe787a2cffd6ae3dbe2888f) traceroot Agent: a detector request, the detectors it read, and its summary of them AI coding agents can also create detectors with the [traceroot CLI](/docs/cli/get-started) (`traceroot detectors create`) or the public API under `/api/v1/public/detectors`. Writes need a signed-in user; a project API key can only read. ## Root cause analysis You set **Root cause analysis** in the **Signals** section, on the **New Detector** page and in the detector's edit panel. It needs **Generate signals** on. * **Automatic** (the default for every new detector): the agent analyzes signals by itself. * **Manual**: the agent runs only when you click **Run root cause analysis** on a signal. Use it to analyze only the problems you choose. Both modes detect and notify the same way. The agent analyzes a signal, not each finding. [Signals](/docs/signals/introduction) explains when it runs. ## View findings Open a detector to see its runs. A run with **Identified** set to Yes is a finding. In the runs table: * Each run links to the trace it checked. * **Identified** shows Yes or No, or a dash if the run failed. * **Signal ID** links to the signal the finding joined. If grouping failed, it says **Not grouped**. * **Agent Run ID** links to the agent's run if the agent analyzed this trace. Otherwise it shows a dash. Open the signal to read its analysis. If the agent analyzed a trace, its analysis opens in the trace's chat panel, and you can ask follow-up questions there. The trace viewer's **Detectors** tab lists each detector that ran on the trace and links to each finding's signal. ## Notifications traceroot tells you when a signal is created or reopened. It collects these signals over each alert window and sends them together in one message, a digest. New findings on a signal it already announced send nothing. In your project, open **Settings → Detectors** and go to **Notifications**: * **Email**: click **Add recipient**, enter an address and click **Save**. * **Alert window**: pick how long traceroot collects signals before it sends a digest (1, 5, 10 or 30 minutes, 1 hour or 2 hours), then click **Save**. The default is 10 minutes. * **Slack**: shows the connected workspace and channel. Click **Connect** or **Manage** to change them. With no email recipient and no Slack channel, traceroot sends nothing. Adding a recipient or channel later does not resend earlier signals. Email and Slack get the same digest, split into **New** and **Reopened**. Each signal shows: * Its title, linked to the signal. * A short summary of what it covers. * Its detector and its number of hits, one per finding. A digest lists up to 45 signals and counts the rest. The email also has a **View signals** button that opens the Signals page. The digest does not include or wait for root cause analysis. Open the signal to read it. ![Slack digest listing new detector signals, each with detector name, hit count and summary](https://mintcdn.com/tracerootai/-OSwxtwDVTPazBuT/images/detector_digest_slack_v2.png?fit=max\&auto=format\&n=-OSwxtwDVTPazBuT\&q=85\&s=c82214d2dd3c939174de142a772c657a) Slack digest: new signals, each with detector, hits and summary ### Connect Slack You connect Slack once per traceroot workspace. traceroot posts to one Slack channel. 1. As a workspace admin, open your workspace settings and go to **Integrations**. From a project, you can also click **Connect** under **Settings → Detectors → Slack**. 2. Click **Connect** and approve traceroot in Slack. 3. Click **Manage → Configure** and choose the channel. Click **Test Connection** to check that it works. > [!NOTE] > > On a self-hosted deployment, signals, root cause analysis and notifications need `OPENAI_API_KEY` on the server. Without it, detectors still record findings. A detector with no model chosen then judges on `claude-haiku-5-5` instead of `gpt-6-luna`, unless the workspace has its own OpenAI key. Email also needs `TRACEROOT_SMTP_URL`. ## Next steps > [card] [Signals](/docs/signals/introduction) > > See how findings are grouped, reviewed and resolved. > [card] [traceroot Agent](/docs/ai-agent/overview) > > Learn how the agent behind root cause analysis works. # Self-Hosting Source: https://traceroot.ai/docs/developer/self-hosting Run traceroot on your own infrastructure > [!NOTE] > > Prefer a managed solution? [traceroot Cloud](https://app.traceroot.ai) is maintained by the traceroot team — no setup required. traceroot is open source and supports self-hosting. Run it on your own infrastructure for full control over your data and deployment. Some features require an enterprise license key. ## Deployment Options ### Local Developer Deployment Best for: contributing to traceroot, local development, and debugging. Starts PostgreSQL, ClickHouse, MinIO, and Redis in Docker, then runs the API, UI, and workers locally via a tmux session. ```bash theme={null} git clone https://github.com/traceroot-ai/traceroot.git cd traceroot make dev ``` ![traceroot local dev environment running in tmux](https://mintcdn.com/tracerootai/nAvKtOoZR-I95teC/images/local_dev_mode_v1.png?fit=max\&auto=format\&n=nAvKtOoZR-I95teC\&q=85\&s=b6bba382d1a142a0661364d888092d0a) See the [README](https://github.com/traceroot-ai/traceroot/blob/main/README.md) and [CONTRIBUTING.md](https://github.com/traceroot-ai/traceroot/blob/main/CONTRIBUTING.md) for details. ### Local Docker Deployment Best for: trying traceroot locally, integration testing, or running a full stack without local dependencies. Runs the entire stack in containers using Docker Compose — no need to install Python, Node, or other runtimes locally. ```bash theme={null} git clone https://github.com/traceroot-ai/traceroot.git cd traceroot make prod ``` See the [README](https://github.com/traceroot-ai/traceroot/blob/main/README.md) for details. ### Credentials `make prod` and `make prod-lite` create a `.env` file and fill in credentials for your install: the database and object store passwords, the auth and internal API secrets, and the key that encrypts stored model provider keys and Slack tokens (`ENCRYPTION_KEY`). Each install gets its own values. To use your own, set them in `.env` before starting; a value copied from `.env.example` is not accepted and the stack stops with a message naming the variable to change. > [!WARNING] > > Keep a backup of `.env`. If `ENCRYPTION_KEY` is lost, stored model provider keys and Slack tokens can no longer be read and have to be entered again. The object store API is published on `127.0.0.1` only. Put a reverse proxy in front of it deliberately if you need access from another machine. ### Upgrading an existing install 1. Pull the latest version and run `make prod-lite` (or `make prod`). This replaces any database or object store password in `.env` that was copied from `.env.example` with a generated one. 2. If the stack stops and names `ENCRYPTION_KEY`, your `.env` still has the key from `.env.example`. Run `make rotate-encryption-key` once. It re-encrypts the stored model provider keys and Slack tokens under a new key in a single transaction, writes the new key to `.env`, and saves the previous one in the `.env.encryption-key.d` directory. If it fails before the database is updated, nothing is changed and you can run it again. If the database was updated but `.env` could not be written, the command says so and keeps the new key in `.env.encryption-key.d/pending`; fix the problem and run it again to finish. It refuses to run while `ENCRYPTION_KEY` is set in your shell environment or `COMPOSE_ENV_FILES` is set, because Docker Compose would then not use the key in `.env`. While a rotation is running, or has not finished, the `make` targets that start the stack wait or refuse, so the services never run with a key the database is being moved away from; to start the stack with `docker compose` yourself, finish the rotation first. 3. Start the stack again with `make prod-lite` (or `make prod`). If you started it with `make prod` before, its tmux session is still there and `make prod` offers to re-attach to it; answer `N` so a new session is started. Re-attaching skips the setup and would leave the stopped services stopped. Rotation stops the web, worker, agent, detector and billing services first. If it fails they stay stopped, and starting is refused until the rotation has finished: fix the problem and run `make rotate-encryption-key` again (to abandon an unfinished rotation that did not change the database, remove `.env.encryption-key.d/pending` and start as in step 3). An install that never set `ENCRYPTION_KEY` has nothing stored under it and simply gets a generated one. ### Production Deployment Best for: production workloads on AWS. Deploy traceroot on Kubernetes using Helm charts and Terraform. Configuration lives in the [`deploy`](https://github.com/traceroot-ai/traceroot/tree/main/deploy) directory. > [!WARNING] > > Terraform deployment is currently experimental. We recommend testing thoroughly before production use. # From the Terminal Source: https://traceroot.ai/docs/evals/cli Read datasets, evaluations, and runs with the traceroot CLI The [traceroot CLI](/docs/cli/get-started) reads the evaluation data the web app shows — your datasets, the versions you published, and the runs that scored them — without leaving the terminal where the eval ran. ```bash theme={null} npm install -g traceroot-cli traceroot login # or use a project API key ``` A project API key is already scoped to one project. A browser login identifies you rather than a project, so pass `--project ` (or set `TRACEROOT_PROJECT_ID`) — `traceroot projects list` shows the ids. ## Find a run Start at the evaluation, narrow to its runs, then read one. Each step prints the id the next one takes. ```bash theme={null} # Which evaluations exist, how many runs each has, and how the latest one ended traceroot evals list # That evaluation's runs, newest first — the RUN ID column is what you read next traceroot evals runs list --evaluation-id ev_… --limit 20 # Only the runs that went wrong traceroot evals runs list --status failed ``` `--status` takes `running`, `completed`, `completed_with_errors`, `failed`, `incomplete`, or `cancelled`. Both lists take `--name ` filtering, case-insensitively, on the evaluation name. A run row is identity and outcome only — which run it is, what it was testing, and when it started. Scores and counts are aggregates over a run's results, so they come from reading the run itself. ## Read a run ```bash theme={null} traceroot evals runs get run_… ``` ``` support-triage · run #12 · completed candidate: git:9f2a1c environment: ci dataset: ds_2a97 @ dsv_1 results 26 reported · 25 scored · 1 task error · 0 scorer errors · 0 not scored METRIC VALUE UNIT CASES accuracy 0.84 — 25 duration 1,205 ms 25 (mean per case) cost 0.0123 $ 25 (mean per case) ``` Each score is a **mean over the results that carried it**, not a headline number for the run: * A **numeric** score's value is its mean, and `CASES` is how many results it was taken over. * A **boolean** score's value is the share that came back true, marked `(share true)`. * A **categorical** score has no mean. It prints `categorical` rather than a number — a label has no average. * **Cost** and **duration** are labelled `(mean per case)`. A run's total cost is a different, and usually much larger, number. ## Read a dataset Datasets read from the dataset down to the cases inside one published version. ```bash theme={null} traceroot datasets list --name triage # newest first, with each one's current version traceroot datasets get ds_… # name, key, description, current version traceroot datasets versions list ds_… # every published version; * marks the current one traceroot datasets versions get dsv_… --limit 50 ``` `datasets versions get` prints the version's cases — case id, input, expected, and the trace a case was captured from where it has one. Versions are immutable, so this is exactly what a run measured: take `dataset_version_id` off a run and read the cases behind it. A dataset with nothing published yet has no version to read. The CLI says so instead of printing an empty table — publish one from the SDK first, as in [Datasets](/docs/evals/datasets). ## Scripting Every command takes `--json` and prints the API response verbatim on stdout. Warnings and counts go to stderr, so a redirect stays valid JSON. ```bash theme={null} traceroot evals runs get run_… --json > run.json traceroot evals runs list --status failed --json | jq -r '.runs[].evaluation_run_id' ``` Commands exit `0` on success and with a class-specific code otherwise — `2` usage, `3` auth, `4` not found, `5` network — so a script can retry a blip and give up on a missing run without parsing prose. ## Two things to know about the output **An absent value prints `—`, never `0`.** A run that is still going has no scored, task-error or scorer-error count yet, and the em dash says exactly that; a `0` would claim a measurement nobody made. `(none)` is a third thing again: a dataset that exists and has no published version. **Each read returns one page.** There is no cursor flag, so when more matched than came back, the command says so on stderr and names the ceiling. The lists return 50 by default and take `--limit` up to 200; `datasets versions get` reads 200 cases by default and takes `--limit` up to 1000. To pull a whole dataset version into code, use `pull_dataset_version()` from the SDK rather than the CLI. ## Command reference | Command | What it does | | :- | :- | | `datasets list` | List the project's datasets, newest first. `--limit `, `--name ` | | `datasets get ` | Show one dataset and the version its cases are read from. | | `datasets versions list ` | List a dataset's published versions with their case counts. `--limit ` | | `datasets versions get ` | Show one version and a page of its cases. `--limit ` (200 by default, up to 1000) | | `evals list` | List evaluations with their run count and latest run. `--limit `, `--name ` | | `evals runs list` | List runs, newest first. `--limit `, `--evaluation-id `, `--status ` | | `evals runs get ` | Show one run: counts, per-scorer means, per-case cost and duration, and its URL. | ## Next steps > [card] [Reading Results](/docs/evals/reading-results) > > The summary and per-case results `evaluate()` returns in process. > [card] [CLI](/docs/cli/get-started) > > Traces, exports, and the rest of what the CLI reads. # Datasets Source: https://traceroot.ai/docs/evals/datasets A named, versioned collection of the cases you evaluate against A dataset is a named collection of **cases** — each with an `input`, an optional `expected` answer, and optional `metadata`. You build it in code, save it to disk, and publish it to the platform when you're ready. ## Create a dataset > [tab] Python > > ```python theme={null} > from traceroot import Dataset > > ds = Dataset("Capitals QA", key="capitals-qa") > ``` > [tab] TypeScript > > ```typescript theme={null} > import { Dataset } from "@traceroot-ai/traceroot"; > > const ds = new Dataset("Capitals QA", null, { key: "capitals-qa" }); > ``` `key` is what identifies the dataset — it defaults to the name. Set it explicitly on any dataset you'll re-run, so you can rename the dataset later without starting a new history. ## Add cases > [tab] Python > > ```python theme={null} > ds.add({"country": "France"}, expected="Paris") > ds.add({"country": "Japan"}, expected="Tokyo", id="case-japan") > > print(len(ds)) > ``` > [tab] TypeScript > > ```typescript theme={null} > ds.add({ country: "France" }, { expected: "Paris" }); > ds.add({ country: "Japan" }, { expected: "Tokyo", id: "case-japan" }); > > console.log(ds.size); > ``` * **`add(input, …)`** — add a case and return it. * **`upsert(case)`** — add or replace by id, so re-running your authoring script never duplicates a case. * **`update(id, …)`** — edit a case in place. * **`archive(id)`** / **`remove(id)`** — retire a case, or delete it outright. A case's id is derived from its content, so adding or reordering cases never renumbers the others. Pass `id=` to use your own identifier instead — a ticket number, or a row id from your warehouse. ## Save and load `save(path)` writes the dataset to disk and `load(path)` reads it back. Use a `.jsonl` path for a format that diffs cleanly in a pull request. > [tab] Python > > ```python theme={null} > ds.save("capitals-qa.jsonl") > ds = Dataset.load("capitals-qa.jsonl") > ``` > [tab] TypeScript > > ```typescript theme={null} > ds.save("capitals-qa.jsonl"); > const ds2 = Dataset.load("capitals-qa.jsonl"); > ``` ## Publish a version `push()` publishes the dataset to the platform as one immutable version. Re-pushing unchanged cases is a no-op; changed cases publish a new version of the same dataset. > [tab] Python > > ```python theme={null} > from traceroot.eval import PlatformDatasetSync > > push = ds.push(PlatformDatasetSync()) > print(push.status, push.version_number) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { PlatformDatasetSync } from "@traceroot-ai/traceroot"; > > const push = await ds.push(new PlatformDatasetSync()); > console.log(push.status, push.versionNumber); > ``` Publishing into a dataset that already exists asks for confirmation on an interactive terminal. Pass `on_existing=lambda info: True` (Python) / `{ onExisting: () => true }` (TypeScript), or set `TRACEROOT_ASSUME_YES=1`, to skip the prompt in CI. A declined prompt raises `DatasetPublishAborted`. You don't need to publish before running an eval — `evaluate()` publishes a local dataset for you and never prompts. See [Running evals](/docs/evals/running-evals). ![Dataset row panel on the Evaluations tab, listing the four runs that used this case](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/dataset_row_evals_v1.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=5b3bf3a5e9af74bebf479ffb87c531fa) Dataset row: Evaluations tab with each run that used the case ## Pull a dataset Pull a published dataset back down as an ordinary local `Dataset` — iterate it, edit it, save it, or evaluate against it. > [tab] Python > > ```python theme={null} > from traceroot import pull_dataset, pull_dataset_version > > ds = pull_dataset("ds_…") # the current version > old = pull_dataset_version("…") # one exact version > ``` > [tab] TypeScript > > ```typescript theme={null} > import { pullDataset, pullDatasetVersion } from "@traceroot-ai/traceroot"; > > const ds = await pullDataset("ds_…"); // the current version > const old = await pullDatasetVersion("…"); // one exact version > ``` To reproduce a past run, pull the version that run recorded (`result.dataset.dataset_version_id`) and re-run it with your own task and scorers. To see what's published without pulling it, the [CLI](/docs/evals/cli) lists datasets, their versions, and the cases in one version: ```bash theme={null} traceroot datasets list traceroot datasets versions list ds_… traceroot datasets versions get dsv_… ``` ## Next steps > [card] [Scorers](/docs/evals/scorers) > > Turn a case's output into a metric. > [card] [Running Evals](/docs/evals/running-evals) > > Point a task and scorers at a dataset and run it. > [card] [From the Terminal](/docs/evals/cli) > > Read datasets and their versions with the CLI. # Run your first agent evaluation Source: https://traceroot.ai/docs/evals/get-started Build a small dataset and score your agent's answers in Python or TypeScript. ## 1. Install the SDK > [tab] Python > > ```bash theme={null} > pip install traceroot > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot > ``` ## 2. Define a dataset A `Dataset` holds your cases. Each case has an `input` and an optional `expected` answer. Set `key` explicitly on any dataset you'll re-run — it's what identifies the dataset, so you can rename it later without starting a new history. > [tab] Python > > ```python theme={null} > from traceroot import Dataset, evaluate > > ds = Dataset("Capitals QA", key="capitals-qa") > ds.add({"country": "France"}, expected="Paris") > ds.add({"country": "Japan"}, expected="Tokyo") > ``` > [tab] TypeScript > > ```typescript theme={null} > import { Dataset, evaluate } from "@traceroot-ai/traceroot"; > > const ds = new Dataset("Capitals QA", null, { key: "capitals-qa" }); > ds.add({ country: "France" }, { expected: "Paris" }); > ds.add({ country: "Japan" }, { expected: "Tokyo" }); > ``` ## 3. Write a task The task is the system under evaluation. It maps an `input` to an `output`, and it may be async. > [tab] Python > > ```python theme={null} > CAPITALS = {"France": "Paris", "Japan": "Tokyo"} > > def task(input): > return CAPITALS.get(input["country"], "") > ``` > [tab] TypeScript > > ```typescript theme={null} > const CAPITALS: Record = { France: "Paris", Japan: "Tokyo" }; > > function task(input: unknown) { > return CAPITALS[(input as { country: string }).country] ?? ""; > } > ``` ## 4. Write a scorer A scorer receives the case's `input`, `output`, `expected`, and `metadata`, and returns a value. The metric takes the function's name. > [tab] Python > > ```python theme={null} > def matches_expected(input, output, expected=None): > return output == expected > ``` > [tab] TypeScript > > ```typescript theme={null} > import type { ScorerContext } from "@traceroot-ai/traceroot"; > > function matchesExpected({ output, expected }: ScorerContext) { > return output === expected; > } > ``` ## 5. Run it `local=True` runs the eval in full and reports nowhere, so you can try it without an API key. > [tab] Python > > ```python theme={null} > result = evaluate( > name="capitals", > dataset=ds, > task=task, > scorers=[matches_expected], > candidate_version="capitals-v1", > local=True, > ) > print(result.summary()) > ``` > [tab] TypeScript > > ```typescript theme={null} > const result = await evaluate({ > name: "capitals", > dataset: ds, > task, > scorers: [matchesExpected], > candidateVersion: "capitals-v1", > local: true, > }); > console.log(result.summary()); > ``` Drop `local` and the run reports to traceroot, using the same credentials as the tracing SDK. Export them in your shell: ```bash theme={null} export TRACEROOT_API_KEY="your-api-key" export TRACEROOT_HOST_URL="https://app.traceroot.ai" # self-hosted: your API's origin, e.g. http://localhost:8000 ``` *(Prefer a `.env` file? Unless your runtime loads it for you, load it yourself — `load_dotenv()` in Python, `config()` from `dotenv` in TypeScript — as the [tracing quickstart](/docs/tracing/get-started) shows.)* `candidate_version` labels what you're testing in each run: a model id, a prompt version, or a git sha. Runs are grouped into one history by `evaluation_key`, which defaults to `name`. ![Evaluations list showing five runs of Support agent golden set with run name, dataset, cost, average cost and duration](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/evals_list_v1.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=53a53ccbb5c04d300e531da8a2f17e17) Evaluations: evaluation name, run name, dataset, cost and duration ## Next steps > [card] [Datasets](/docs/evals/datasets) > > Author, version, and publish datasets. > [card] [Scorers](/docs/evals/scorers) > > Code scorers and LLM-as-a-judge. > [card] [Running Evals](/docs/evals/running-evals) > > Every option of `evaluate()`. > [card] [Reading Results](/docs/evals/reading-results) > > Per-metric summaries and per-case results. ## Run the example Clone the repo and run a complete tool-agent eval end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/agent-eval) > > Run the Python example > [card] [TypeScript](https://github.com/traceroot-ai/traceroot/tree/main/examples/typescript/agent-eval) > > Run the TypeScript example # Evaluate AI agents with datasets and scorers Source: https://traceroot.ai/docs/evals/introduction Inspect the trace for each test case and compare scores across runs. An **evaluation** runs your system against a dataset of cases and scores each result. In traceroot every case runs as its own trace, so you inspect an eval with the same span timeline, token, and cost views you already use for tracing. Each scorer emits a metric, and you read a run per metric — there is no single headline score. The SDK is available in **Python** and **TypeScript**, with the same API in both. ## What you write * A **dataset** — the cases to run, each with an `input` and an optional `expected` answer. * A **task** — your system under evaluation, mapping an `input` to an `output`. * One or more **scorers** — functions that turn a case's result into a metric. Then `evaluate()` runs the task over every case, scores each result, and reports the run. ## Where to start > [card] [Get Started](/docs/evals/get-started) > > Author a dataset, write a scorer, and run your first eval. > [card] [Datasets](/docs/evals/datasets) > > Author, version, and publish the cases you run against. > [card] [Scorers](/docs/evals/scorers) > > Code scorers and LLM-as-a-judge. > [card] [Running Evals](/docs/evals/running-evals) > > `evaluate()` — the task, the scorers, and where a run reports. > [card] [Reading Results](/docs/evals/reading-results) > > Per-metric summaries and per-case results. > [card] [From the Terminal](/docs/evals/cli) > > Read datasets, evaluations, and past runs with the CLI. # Reading Results Source: https://traceroot.ai/docs/evals/reading-results Read the summary and per-case results an eval run returns `evaluate()` returns an `EvalRunResult` — a summary per metric, one result per case, and where the run reported. ## Summary `summary()` prints one line per metric: ``` matches_expected: mean=0.75 pass=3/4 count=4 ``` A boolean score passes or fails on its own value. A numeric score needs a `threshold` to be judged — without one it's averaged, and `pass=k/n` is left off. > [tab] Python > > ```python theme={null} > result = evaluate(name="weather", dataset=ds, task=task, scorers=[reports_temp]) > print(result.summary()) > ``` > [tab] TypeScript > > ```typescript theme={null} > const result = await evaluate({ name: "weather", dataset: ds, task, scorers: [reportsTemp] }); > console.log(result.summary()); > ``` Token counts and cost aren't in the local summary — the platform prices them from the run's spans and shows them in the UI. ## Per-case results > [tab] Python > > ```python theme={null} > for item in result.results: > print(item.case_id, item.output, item.scores) > ``` > [tab] TypeScript > > ```typescript theme={null} > for (const item of result.itemResults) { > console.log(item.caseId, item.output, item.scores); > } > ``` Each item carries `case_id`, `input`, `output`, `expected`, `scores`, `trace_id`, `duration_ms`, and any `error` or `scorer_errors`. A case that hit an error is `errored`; use `case_status(item)` / `caseStatus(item)` to check. The run also exposes counts (`case_count`, `errored`, `not_scored`), the `candidate_version` it was labelled with, `upload_state` with the run's `dashboard_url`, and `dataset` — the exact dataset version the run measured. ![Evaluation run results table with each case's input, output, expected value and scorer columns](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/eval_run_v1.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=d009c4f45f8bb55c6d695f71f69dab41) Evaluation run: each case's input, output, expected answer and scores ## Upload later Save a run and upload it afterwards, for when the run happens somewhere without credentials. > [tab] Python > > ```python theme={null} > from traceroot import EvalRunResult > > result.save("run.json") > > later = EvalRunResult.load("run.json") > later.upload() > ``` > [tab] TypeScript > > ```typescript theme={null} > import { EvalRunResult } from "@traceroot-ai/traceroot"; > > result.save("run.json"); > > const later = EvalRunResult.load("run.json"); > await later.upload(); > ``` `upload()` rebuilds the reporting destination from the saved run, and keeps its scorers' thresholds — so a later upload records the same pass and fail results as the original run. ## Reading a run later `EvalRunResult` is what the process that ran the eval holds. Once the run has reported, the [CLI](/docs/evals/cli) reads it back from anywhere — including a run somebody else started. ```bash theme={null} traceroot evals runs list --evaluation-id ev_… # find the run id traceroot evals runs get run_… # counts, per-scorer means, cost and duration ``` ## Next steps > [card] [From the Terminal](/docs/evals/cli) > > Read datasets, evaluations, and past runs with the CLI. > [card] [Running Evals](/docs/evals/running-evals) > > Every option of `evaluate()`, and where a run reports. > [card] [Datasets](/docs/evals/datasets) > > Pull a dataset version to reproduce a past run. # Running Evals Source: https://traceroot.ai/docs/evals/running-evals Run a dataset against your system with evaluate() `evaluate()` runs your system against a dataset, scores every case, and reports the run. Each case runs as its own trace, so you can open any case and see exactly what your system did. ## Parameters Call `evaluate()` with keyword arguments. TypeScript takes the same names in camelCase (`candidateVersion`, `evaluationKey`, `maxConcurrency`) as a single options object. | Parameter | Type | Default | Description | | - | - | - | - | | `name` | `str` | required | The evaluation's name | | `dataset` | `Dataset` or `list` | required | The [dataset](/docs/evals/datasets) to run against, or an inline list of cases | | `task` | `callable` | required | Your system, mapping an input to an output. May be async | | `scorers` | `list` | required | The [scorers](/docs/evals/scorers) to grade each case with | | `candidate_version` | `str` | `"sdk"` | Labels what you're testing (model id, prompt version, git sha). Shown next to the run number in the UI | | `evaluation_key` | `str` | `name` | The stable identity runs are grouped under. Set it to keep one history across a rename, or to group the Python and TypeScript runs of one evaluation | | `local` | `bool` | `False` | Run in full but report nowhere. Cannot be combined with `transport` | | `transport` | `EvalTransport` | `None` | An explicit destination for the run. Pass `FakeTransport()` to record what would have been sent, without any HTTP | | `select` | `callable` | `None` | A predicate to run a subset of cases without editing the dataset | | `timeout` | `float` | `None` | Seconds allowed per case. A timeout fails that case only, not the run | | `progress` | `bool` | auto | The live progress bar. On for an interactive terminal; pass `False` to silence it in CI | | `max_concurrency` | `int` | `10` | How many cases run in parallel | An inline list of cases runs fine, but only a `Dataset` can be reported to the platform — pass `local=True` when running against a list. Leave `local` and `transport` unset and the run reports to traceroot when credentials resolve. Python has both `evaluate` and `evaluate_async`. In TypeScript, `evaluate` is async — always `await` it. ## Example > [tab] Python > > ```python theme={null} > from traceroot import Dataset, evaluate > > ds = Dataset("weather", key="weather") > ds.add({"city": "SF"}, expected="68°F") > > def task(input): > return look_up_weather(input["city"]) > > def reports_temp(input, output, expected=None): > return expected is not None and expected in (output or "") > > result = evaluate( > name="weather", > dataset=ds, > task=task, > scorers=[reports_temp], > candidate_version="weather-v1", > evaluation_key="weather-eval", > ) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { Dataset, evaluate, type ScorerContext } from "@traceroot-ai/traceroot"; > > const ds = new Dataset("weather", null, { key: "weather" }); > ds.add({ city: "SF" }, { expected: "68°F" }); > > async function task(input: unknown) { > return lookUpWeather((input as { city: string }).city); > } > > function reportsTemp({ output, expected }: ScorerContext) { > return String(output ?? "").includes(String(expected)); > } > > const result = await evaluate({ > name: "weather", > dataset: ds, > task, > scorers: [reportsTemp], > candidateVersion: "weather-v1", > evaluationKey: "weather-eval", > }); > ``` A local `Dataset` is published for you as part of the run, so you don't need a separate publish step. On an interactive terminal the summary prints itself; piped and CI output stays clean. ## Next steps > [card] [Reading Results](/docs/evals/reading-results) > > Read the summary and the per-case results `evaluate()` returns. > [card] [Datasets](/docs/evals/datasets) > > Author, version, publish, and pull datasets. # Scorers Source: https://traceroot.ai/docs/evals/scorers Turn a case's output into a metric A scorer receives a case's `input`, `output`, `expected`, and `metadata`, and returns a metric. There are two kinds: **code scorers**, which you write as a function, and **LLM-as-a-judge** scorers, which grade the output with a model. ## Code scorers Any function is a scorer. The metric takes the function's name. > [tab] Python > > ```python theme={null} > def reports_temp(input, output, expected=None, metadata=None): > return expected is not None and expected in (output or "") > ``` > [tab] TypeScript > > ```typescript theme={null} > import type { ScorerContext } from "@traceroot-ai/traceroot"; > > function reportsTemp({ output, expected }: ScorerContext) { > return typeof output === "string" && output.includes(String(expected)); > } > ``` Return a boolean, a number, or any of the [return values](#what-a-scorer-can-return) below. A boolean passes or fails on its own value; a number is averaged until you give it a threshold. ### Adding a threshold Wrap the function in `Scorer.code` to give a numeric metric a pass mark, so the platform can mark each score as passing or failing and compare runs. > [tab] Python > > ```python theme={null} > from traceroot import Scorer > > @Scorer.code( > key="covers_cities", > value_type="numeric", # "numeric" | "boolean" | "categorical" > direction="higher_is_better", # "higher_is_better" | "lower_is_better" | "none" > threshold=1.0, > ) > def covers_cities(ctx): > cities = ctx.input["cities"] > output = (ctx.output or "").lower() > return sum(c.lower() in output for c in cities) / len(cities) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { Scorer, type ScorerContext } from "@traceroot-ai/traceroot"; > > const coversCities = Scorer.code( > { > key: "covers_cities", > name: "covers_cities", > valueType: "numeric", // "numeric" | "boolean" | "categorical" > direction: "higher_is_better", // "higher_is_better" | "lower_is_better" | "none" > threshold: 1.0, > }, > (ctx: ScorerContext) => { > const cities = (ctx.input as { cities: string[] }).cities; > const output = String(ctx.output ?? "").toLowerCase(); > return cities.filter((c) => output.includes(c.toLowerCase())).length / cities.length; > }, > ); > ``` * **`key`** identifies the scorer. Use the same `key` in Python and TypeScript for the same scorer. * **`threshold` and `direction`** decide whether a numeric score passes. * The metric takes the scorer's name. If you return a `Score` with a different name, give it that scorer's name too — otherwise the threshold can't be matched to it. ## LLM-as-a-judge For a judge, the model and the prompt are the scorer. Write the prompt with `{{input}}`, `{{output}}`, and `{{expected}}` placeholders, filled in per case. > [tab] Python > > ```python theme={null} > from traceroot import Scorer > > is_grounded = Scorer.llm_judge( > name="is_grounded", > model="claude-haiku-5-5", > messages=[ > {"role": "system", "content": "Reply 1.0 if the answer states concrete values, else 0.0."}, > {"role": "user", "content": "ANSWER:\n{{output}}"}, > ], > value_type="numeric", > threshold=1.0, > ) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { Scorer } from "@traceroot-ai/traceroot"; > > const isGrounded = Scorer.llmJudge({ > name: "is_grounded", > model: "claude-haiku-5-5", > messages: [ > { role: "system", content: "Reply 1.0 if the answer states concrete values, else 0.0." }, > { role: "user", content: "ANSWER:\n{{output}}" }, > ], > valueType: "numeric", > threshold: 1.0, > }); > ``` `rubric="…"` is a shorthand when you only need a single grading instruction. ### Building the prompt from the case To fill in your own placeholders, pass a function that returns the template values. It returns variables for the prompt, never a score. > [tab] Python > > ```python theme={null} > @Scorer.llm_judge( > name="comparison_present", > model="claude-haiku-5-5", > messages=[{"role": "user", "content": "Does {{answer}} compare {{cities}}? Reply 1.0 or 0.0."}], > value_type="numeric", > threshold=1.0, > ) > def comparison_present(ctx): > return {"answer": ctx.output, "cities": " and ".join(ctx.input["cities"])} > ``` > [tab] TypeScript > > ```typescript theme={null} > const comparisonPresent = Scorer.llmJudge( > { > name: "comparison_present", > model: "claude-haiku-5-5", > messages: [{ role: "user", content: "Does {{answer}} compare {{cities}}? Reply 1.0 or 0.0." }], > valueType: "numeric", > threshold: 1.0, > }, > (ctx) => ({ > answer: ctx.output, > cities: (ctx.input as { cities: string[] }).cities.join(" and "), > }), > ); > ``` Pass `complete=` (Python) / `complete:` (TypeScript) — a function taking the model and messages and returning a string — to stub the model call, so judges run in tests and CI without an API key. ## What a scorer can return * a **boolean** or **number**, * a **`Score`** — `Score(name, value, comment=None, metadata=None)`, * a **list of `Score`s**, or a **`{metric: value}`** map, for a scorer that emits several metrics, * a **`DeferredScore`** — a score awaiting human or async review. It's recorded as pending, never as a zero. ## Next steps > [card] [Running Evals](/docs/evals/running-evals) > > Point a task and your scorers at a dataset and run the eval. > [card] [Reading Results](/docs/evals/reading-results) > > Per-metric averages, pass rates, and case status. # Coding agents Source: https://traceroot.ai/docs/getting-started/coding-agents Let your coding agent set up traceroot, search these docs, and work with your project Your coding agent can set up traceroot, look up these docs, work with your project from the CLI, and send its own sessions to traceroot. ## Set up tracing You need Node.js 20.12 or later and a Python or TypeScript/JavaScript application. If you don't have a traceroot account yet, you can create one when you sign in. Paste this into Claude Code, Codex, or another coding agent, from the root of your repository: ```text Setup prompt wrap theme={null} Set up traceroot tracing in this repo with the traceroot setup wizard. 1. From the repo root, run: npx -y traceroot-cli@latest setup --no-input - If it says no credential is configured, run npx -y traceroot-cli@latest login (allow up to 10 minutes), show me the URL and code it prints, and wait while I sign in. Then run setup again. - If it stops for any other reason, follow the remedy it prints. Ask me before choosing a project. 2. Setup writes a project API key to .env.local or .env and an instrumentation task to .traceroot/prompts/setup-instrument.md. Carry out that task yourself, in this session. 3. When the task is done, don't stop. Rerun the same setup command, keeping all flags, and add --resume. You're done when it prints a link to the first trace. Don't print the API key or commit the file that holds it. Reference: https://traceroot.ai/docs/cli/get-started.md ``` Your agent runs setup, shows you a sign-in link and code if it needs one, adds the traceroot SDK, and finishes with a link to your first trace. Setup never reverts your agent's edits, so review them with `git status` and `git diff`. To run setup in your terminal instead, see [Tracing](/docs/tracing/get-started). If you're self-hosting, start with [Self-hosted traceroot](/docs/tracing/get-started#self-hosted-traceroot). ## Skills Skills teach your agent how to work with traceroot. Setup installs its own instrumentation skill, so you don't need one to set up tracing. | Skill | Use it to | | :- | :- | | `traceroot-cli` | Read traces, evaluations, datasets, and detector findings from the terminal | | `traceroot-eval` | Write and run offline evaluations | | `traceroot-quickstart` | Send a first test trace without an existing LLM application | Install one with: ```bash theme={null} npx skills add traceroot-ai/traceroot-skills --skill traceroot-cli ``` `npx skills` is a third-party CLI that needs Node.js 22.20 or later. The `traceroot-cli` skill expects the `traceroot` command, so install it with `npm install -g traceroot-cli`. ## Docs MCP `https://traceroot.ai/docs/mcp` lets your agent search and read these docs, with no API key. For your traces, use [the CLI](#the-cli). > [tab] Claude Code > > ```bash theme={null} > claude mcp add --transport http traceroot-docs https://traceroot.ai/docs/mcp > ``` > [tab] Codex > > ```bash theme={null} > codex mcp add traceroot-docs --url https://traceroot.ai/docs/mcp > ``` > [tab] Cursor > > Add this to `.cursor/mcp.json`: > > ```json theme={null} > { > "mcpServers": { > "traceroot-docs": { > "url": "https://traceroot.ai/docs/mcp" > } > } > } > ``` ## Markdown docs For any page as Markdown, add `.md` to its URL. Requests that send `Accept: text/markdown` get Markdown too. * [llms.txt](https://traceroot.ai/docs/llms.txt) lists every page. Start here. * [llms-full.txt](https://traceroot.ai/docs/llms-full.txt) holds every page in one file, about 300 KB. * [skill.md](https://traceroot.ai/docs/skill.md) is a short guide for agents: setup, common mistakes, and where to look next. ## The CLI Your agent works with your project through the traceroot CLI: traces, detector findings, alerts, dashboards, evaluations, and SQL. Run these from the directory you ran setup in. Anywhere else, sign in with `traceroot login` and [scope to a project](/docs/cli/get-started#scope-to-a-project) first. Without a global install, replace `traceroot` with `npx -y traceroot-cli@latest`. ```bash theme={null} # Recent traces traceroot traces list --since 1h --limit 10 --json # One trace, with span input and output traceroot traces get --fields full --json # Recent detector findings traceroot findings list --since 24h --json # Recent evaluation runs traceroot evals runs list --json ``` Without `--fields full`, `traces get --json` leaves span input and output `null`. Run `traceroot --help` for every flag, and `traceroot doctor` to check your setup. [Output and exit codes](/docs/cli/get-started#output-and-exit-codes) covers what scripts and agents can rely on. ## Trace your coding agent Plugins for [Claude Code](/docs/integrations/claude-code), [Codex](/docs/integrations/codex), and [pi](/docs/integrations/pi-cli) send your coding agent's own sessions to traceroot, which is separate from tracing your application. Sessions can include your prompts, tool input and output, commands, and file contents. If you export `TRACEROOT_API_KEY` for a plugin, it also applies to anything else you or your agent start from that shell, and it takes priority over the key in a repository's env file. ## Next steps > [card] [Tracing](/docs/tracing/get-started) > > Run setup yourself, add the SDK by hand, and troubleshoot. > [card] [CLI](/docs/cli/get-started) > > Work with traces, findings, alerts, and evaluations from the terminal. # Workspaces, projects, and access Source: https://traceroot.ai/docs/getting-started/concepts Organize trace data, invite teammates, and choose workspace roles ## Workspaces and projects A workspace brings together your team and its projects. Membership and roles belong to the workspace; your role applies to its projects. Workspace settings include members and integrations such as Slack and model providers. A project groups an application's traces, detectors, evaluations, datasets, dashboards, and alerts. Choose the workspace and project you want to work in before opening a product page. Project API keys are scoped to one project. Create a key in the project's settings and use it with the SDK for the application that should send data to that project. A key for one project does not grant access to another project's data. Keep keys secret. ## Environments An environment labels traffic within a project, for example `production` or `staging`. It is a field on trace data, rather than a separate workspace, project, or permission boundary. Use environment filters to focus on one deployment. The deployment label is carried by the `traceroot.environment` span attribute. With the [TypeScript SDK](/docs/tracing/typescript-sdk), set `TRACEROOT_ENVIRONMENT` to label the deployment. ## Invite a teammate 1. As a workspace ADMIN, open the workspace's **Settings** and select **Members**. 2. Click **Invite member**, enter the teammate's email address, and choose VIEWER, MEMBER, or ADMIN. 3. Send the invitation. The teammate receives an email linking to the invite page. 4. The teammate opens the link and signs in with the invited email address to accept it. Admins can manage pending invitations and change members' roles from the Members tab. Invitations and acceptance are subject to the workspace's seat limit. If acceptance reports that the workspace is full, ask an admin to free a seat or change the plan. ## Roles Roles are ordered VIEWER, MEMBER, ADMIN. A higher role includes the permissions of the roles below it. The table describes signed-in workspace members; project API keys use their own credential scope. | Action | VIEWER | MEMBER | ADMIN | | :- | :-: | :-: | :-: | | View project traces, detectors, dashboards, and alerts | Yes | Yes | Yes | | Create and edit detectors | No | Yes | Yes | | Create and edit dashboards and widgets | No | Yes | Yes | | Create, edit, pause, and resume alerts | No | Yes | Yes | | Create a project | No | Yes | Yes | | Create and edit project API keys | No | Yes | Yes | | Delete project API keys | No | No | Yes | | Invite members, change roles, or remove members | No | No | Yes | | Rename a project or change its retention settings | No | No | Yes | | Delete a project | No | No | Yes | | Rename or delete a workspace | No | No | Yes | If an API or CLI write returns a permission error, check the role on the workspace containing the project. Having MEMBER access in one workspace does not make you an ADMIN in another. Ask a workspace admin to perform the action or update your role. ## Choose your next workflow * [Tracing](/docs/tracing/get-started): send your first trace. * [Evaluations](/docs/evals/get-started): test an agent against a dataset. * [Dashboards](/docs/dashboards/get-started): explore project metrics. * [Alerts](/docs/alerts/overview): monitor a metric threshold. # How traceroot works Source: https://traceroot.ai/docs/getting-started/introduction Use production traces to find out why an agent failed and open a fix on GitHub. Find the patterns in your agent's production issues. Ship verified fixes with your coding agent. Instrument your code with [traceroot](https://github.com/traceroot-ai/traceroot) to see every LLM call, tool invocation, and agent decision. Run detectors to catch failures and hallucinations the moment they happen. Then let traceroot's AI agent analyze findings, debug failures and create fix PRs automatically. ## Key Capabilities > [card] [Tracing](/docs/tracing/introduction) > > Capture every LLM call, tool use, and agent step with OpenTelemetry-based instrumentation. > [card] [Detectors](/docs/detectors/introduction) > > Continuously evaluate every trace with an LLM; get alerted the moment your agent fails, hallucinates, or drifts. > [card] [traceroot Agent](/docs/ai-agent/overview) > > Root cause analysis and automated fix PRs — powered by the traceroot Agent with access to your source code. > [card] [Evaluations](/docs/evals/introduction) > > Test changes against versioned datasets and compare scores across evaluation runs. > [card] [Dashboards](/docs/dashboards/get-started) > > Explore latency, cost, and error metrics with filtered, reusable widgets. > [card] [Alerts](/docs/alerts/overview) > > Monitor metric thresholds and send notifications to your team's Slack channel. Start with the [quickstart](/docs/getting-started/quickstart), or learn how [workspaces, projects, and roles](/docs/getting-started/concepts) organize access to your data. # Quickstart Source: https://traceroot.ai/docs/getting-started/quickstart Get up and running with traceroot > [card] [Tracing](/docs/tracing/get-started) > > Capture every LLM call, tool use, and agent step with OpenTelemetry-based instrumentation. > [card] [Detectors](/docs/detectors/introduction#how-to-create-a-detector) > > Create your first detector and start surfacing failures, hallucinations, and missed tasks. > [card] [Self-Hosting](/docs/developer/self-hosting) > > Run traceroot on your own infrastructure. > [card] [CLI](/docs/cli/get-started) > > List, inspect, and export your traces from the terminal. > [card] [traceroot Agent](/docs/ai-agent/overview) > > Investigate traces and detector findings with an assistant that can work with your source code. > [card] [Evaluations](/docs/evals/introduction) > > Run your AI system against a dataset and score every case. > [card] [Datasets](/docs/evals/datasets) > > Build, version, and publish the cases you evaluate against. > [card] [Dashboards](/docs/dashboards/get-started) > > Track your project's latency, cost, errors, and other metrics with widgets. > [card] [Alerts](/docs/alerts/overview) > > Notify your team when a metric crosses a threshold. Read [Workspaces, projects, and access](/docs/getting-started/concepts) to learn how to organize your data, invite teammates, and choose their roles. # Agno Source: https://traceroot.ai/docs/integrations/agno Auto-instrument Agno agents, tools, and multi-step reasoning Automatically capture agent runs, tool calls, and reasoning steps within the [Agno](https://github.com/agno-agi/agno) framework. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot agno anthropic duckduckgo-search yfinance > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.AGNO]) > ``` ## Usage Once initialized, all Agno agent runs and tool invocations are captured automatically: ```python theme={null} import traceroot from traceroot import Integration from agno.agent import Agent from agno.models.anthropic import Claude from agno.tools.duckduckgo import DuckDuckGoTools from agno.tools.yfinance import YFinanceTools traceroot.initialize(integrations=[Integration.AGNO]) agent = Agent( model=Claude(id="claude-sonnet-4-20250514"), tools=[ YFinanceTools(), DuckDuckGoTools(), ], instructions=[ "Use YFinance for stock prices and fundamentals.", "Use DuckDuckGo for general web searches.", "Always cite your sources.", ], markdown=True, ) # The entire agent run, including every tool call, is traced agent.print_response( "What is the current stock price of NVDA and what are its fundamentals?", stream=False, ) traceroot.flush() ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `Agent.run`. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Agent runs | Each `agent.run()` / `agent.print_response()` invocation | | Reasoning steps | Individual ReAct-style steps within the agent loop | | Tool calls | Each tool invocation with input arguments and results | | LLM calls | Raw completion requests to the underlying provider | | Tokens & Cost | Aggregated token usage; cost when the model ID matches traceroot's pricing table | | Latency | Duration per agent run and per span | ## Limitations None known. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/agno-tool-agent) > > Run the Python example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # Anthropic Source: https://traceroot.ai/docs/integrations/anthropic Auto-instrument Anthropic API calls Automatically capture all Anthropic Messages API calls. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot anthropic > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot @anthropic-ai/sdk > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.ANTHROPIC]) > ``` > [tab] TypeScript > > ```typescript theme={null} > import Anthropic from '@anthropic-ai/sdk'; > import * as anthropicSDK from '@anthropic-ai/sdk'; > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize({ instrumentModules: { anthropic: anthropicSDK } }); > > const client = new Anthropic(); > ``` ## Usage Once initialized, all Anthropic calls are captured automatically: > [tab] Python > > ```python theme={null} > import anthropic > > client = anthropic.Anthropic() > > # This call is automatically traced > response = client.messages.create( > model="claude-sonnet-4-20250514", > max_tokens=1024, > messages=[ > {"role": "user", "content": "What is the capital of France?"}, > ], > ) > > print(response.content[0].text) > ``` > [tab] TypeScript > > ```typescript theme={null} > import Anthropic from '@anthropic-ai/sdk'; > > const client = new Anthropic(); > > // This call is automatically traced > const response = await client.messages.create({ > model: 'claude-sonnet-4-20250514', > max_tokens: 1024, > messages: [ > { role: 'user', content: 'What is the capital of France?' }, > ], > }); > > console.log(response.content[0].text); > ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `messages.create` (`Anthropic Messages` on TypeScript). 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Model | `claude-sonnet-4-20250514`, `claude-haiku-4-5-20251001`, etc. | | Messages | Input messages array | | Response | Completion content | | Tokens | Input and output tokens | | Cost | Calculated when the model ID matches traceroot's pricing table | | Latency | Request duration | ## Limitations None known. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/anthropic-tool-agent) > > Run the Python example > [card] [TypeScript](https://github.com/traceroot-ai/traceroot/tree/main/examples/typescript/anthropic) > > Run the TypeScript example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [TypeScript SDK Reference](/docs/tracing/typescript-sdk) > > Explore the full TypeScript SDK reference and configuration options. # AutoGen Source: https://traceroot.ai/docs/integrations/autogen Auto-instrument AutoGen multi-agent conversations and tool calls Automatically capture agent loops, message histories, and tool executions within the AutoGen framework. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot "ag2[gemini]>=0.14.0,<1.0" > ``` ## Set up This integration instruments [AG2](https://github.com/ag2ai/ag2), the community-maintained continuation that provides the `autogen` package. Microsoft's newer `autogen-agentchat` package uses a different API and is not currently supported by this integration. To capture the complete picture—including both the agent orchestrations and the underlying token costs—we highly recommend initializing both the AG2 integration and your specific LLM provider (e.g., Google GenAI, OpenAI, Anthropic). > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > # Initialize AutoGen alongside your LLM provider to capture tokens and costs > traceroot.initialize(integrations=[ > Integration.AUTOGEN, > Integration.GOOGLE_GENAI # Or OPENAI, ANTHROPIC, etc. > ]) > ``` ## Usage Once initialized, agent conversations and tool calls are captured automatically: ```python theme={null} import os import autogen from typing import Annotated from autogen import register_function llm_config = { "config_list": [{ "model": "gemini-2.5-flash", "api_key": os.environ["GEMINI_API_KEY"], "api_type": "google" }] } assistant = autogen.AssistantAgent( name="assistant", llm_config=llm_config ) user_proxy = autogen.UserProxyAgent( name="user_proxy", human_input_mode="NEVER", max_consecutive_auto_reply=5 ) def get_weather(city: Annotated[str, "City name"]) -> str: """Get current weather for a city.""" return f"The weather in {city} is 72 degrees and sunny." def get_forecast(city: Annotated[str, "City name"], days: Annotated[int, "Number of days"]) -> str: """Get a multi-day weather forecast for a city.""" return f"{days}-day forecast for {city}: mostly sunny with highs around 70-75°F." # Register tools — caller decides when to use them, executor runs the function register_function( get_weather, caller=assistant, executor=user_proxy, name="get_weather", description="Get current weather for a city", ) register_function( get_forecast, caller=assistant, executor=user_proxy, name="get_forecast", description="Get a multi-day weather forecast for a city", ) # The entire conversation hierarchy is automatically traced user_proxy.initiate_chat( assistant, message="What's the weather in Tokyo today, and what does the 3-day forecast look like?", ) ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `user_proxy.initiate_chat`, named after the agent that opened the chat. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Conversation Loop | The overarching `initiate_chat` session | | Agent Steps | Individual spans for each `AssistantAgent` or `UserProxyAgent` turn | | Messages | Full chat history, input messages, and agent replies | | Tool calls | Function names, input arguments, and execution outputs | | LLM calls | Raw completion requests to the provider | | Tokens & Cost | Aggregated usage; cost when the model ID matches traceroot's pricing table | ## Limitations * AutoGen (AG2) requires initializing both `Integration.AUTOGEN` and your underlying LLM provider integration (such as `Integration.GOOGLE_GENAI` or `Integration.OPENAI`) to capture full token usage and pricing. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/autogen-tool-agent) > > Run the Python example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # Claude Agent SDK Source: https://traceroot.ai/docs/integrations/claude-agent-sdk Auto-instrument the Claude Agent SDK for agent and tool tracing Automatically capture agent invocations, subagent delegations, tool calls, and token usage from the Claude Agent SDK (Claude Code as a library). ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot claude-agent-sdk > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot @anthropic-ai/claude-agent-sdk > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.CLAUDE_AGENT_SDK]) > ``` > [tab] TypeScript > > ```typescript theme={null} > import * as claudeAgentSDK from '@anthropic-ai/claude-agent-sdk'; > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize({ > instrumentModules: { claudeAgentSDK }, > }); > ``` ## Usage > [tab] Python > > ```python theme={null} > from claude_agent_sdk import query, ClaudeAgentOptions, AgentDefinition > > # Define subagents > researcher = AgentDefinition( > description="Research specialist for gathering information.", > prompt="You are a research specialist. Use WebSearch to find info.", > tools=["WebSearch"], > model="haiku", > ) > > # Run the agent with subagents > async for message in query( > prompt="Research the latest trends in AI observability", > options=ClaudeAgentOptions( > allowed_tools=["Agent", "Bash", "Read", "Glob", "Grep"], > agents={"researcher": researcher}, > ), > ): > if hasattr(message, "result"): > print(message.result) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { query, AgentDefinition } from '@anthropic-ai/claude-agent-sdk'; > > const researcher: AgentDefinition = { > description: 'Research specialist for gathering information.', > prompt: 'You are a research specialist. Use WebSearch to find info.', > tools: ['WebSearch'], > model: 'haiku', > }; > > async function main() { > for await (const message of query({ > prompt: 'Research the latest trends in AI observability', > options: { > allowedTools: ['Agent', 'Bash', 'Read', 'Glob', 'Grep'], > agents: { researcher }, > }, > })) { > if ('result' in message) { > console.log(message.result); > } > } > } > main().catch(console.error); > ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `ClaudeAgent.query`. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Agent queries | Each `query()` call as a top-level agent span | | Subagent invocations | Each subagent delegation via the Agent tool | | Tool calls | Built-in tools (Read, Write, Bash, Glob, Grep, WebSearch) | | Token usage | Aggregated input/output tokens per query | | Cost | Aggregated when the model ID matches traceroot's pricing table | | Model name | The model used (e.g., claude-sonnet-4-6, claude-haiku-4-5) | ## Limitations None known. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/claude-agent-sdk) > > Run the Python example > [card] [TypeScript](https://github.com/traceroot-ai/traceroot/tree/main/examples/typescript/claude-agent-sdk) > > Run the TypeScript example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [TypeScript SDK Reference](/docs/tracing/typescript-sdk) > > Explore the full TypeScript SDK reference and configuration options. # Claude Code Source: https://traceroot.ai/docs/integrations/claude-code Trace Claude Code sessions with the traceroot plugin Trace every Claude Code request — model calls, tool calls, sub-agents and token usage — with the [traceroot Claude Code plugin](https://github.com/traceroot-ai/traceroot-claude-code-plugin). ## Install The plugin runs its hook with `uv` (recommended) or Python 3.11+ with the `traceroot` package installed. ```bash theme={null} claude plugin marketplace add traceroot-ai/traceroot-claude-code-plugin claude plugin install traceroot-observability@traceroot-claude-code-plugin ``` ## Set up Claude Code asks for the plugin settings when you install it and stores the API key in your OS keychain. To check or change them later, run `/plugin` in Claude Code, or from a terminal: ```bash theme={null} # Show the current settings and which are unset claude plugin configure traceroot-observability@traceroot-claude-code-plugin # Self-hosted only: point the plugin at your deployment echo '{"TRACEROOT_HOST_URL": "https://your-traceroot.example.com"}' \ | claude plugin configure traceroot-observability@traceroot-claude-code-plugin --values-stdin ``` | Setting | Option | Default | Description | | - | - | - | - | | API key | `TRACEROOT_API_KEY` | — | Required. Your traceroot API key. | | Host | `TRACEROOT_HOST_URL` | `https://app.traceroot.ai` | Set only when self-hosting. | | Max chars per field | `TRACEROOT_PLUGIN_MAX_CHARS` | `20000` | Longer inputs and outputs are shortened in the middle. | | Debug logging | `TRACEROOT_PLUGIN_DEBUG` | `false` | Writes diagnostics to `~/.claude/state/traceroot/debug.log`. | An environment variable with the same name is used only when the plugin setting is empty. Settings with a default always have a value, so an `export` won't change them; set them in the plugin settings instead. Restart Claude Code after changing them. ## Usage Use Claude Code as usual. Each request you send becomes one trace, sent when Claude Code finishes responding (after any sub-agents report back). Traces from the same session share a session ID. ``` Claude Code Turn # AGENT span — one per request ├── claude-opus-… # LLM span — tokens, cache reads and writes │ ├── Bash # TOOL span — input and output │ └── Edit └── Agent # TOOL span — a sub-agent's model and tool calls ``` ## Verify Send one request in Claude Code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `Claude Code Turn`. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Requests | One trace per request, grouped by Claude Code session | | Model calls | Model id, prompt and response, extended thinking | | Tokens | Input, output, cache read and cache write counts | | Tool calls | Tool name, input and output; failed tools are marked as errors | | Sub-agents | Each sub-agent's model and tool calls under its Agent span | | Git | Repository and ref of the working directory | Prompts, tool inputs and tool outputs are sent as they are, including file contents and command output. Nothing is redacted. See [Data Capture and Privacy](/docs/tracing/data-capture#what-is-captured-by-default). ## Limitations * No user ID is recorded, so traces don't appear on the **Users** page. * If no trace arrives, turn on **Debug logging** in the plugin settings and check `~/.claude/state/traceroot/debug.log`. Without an API key the plugin does nothing. ## Run the example > [card] [traceroot-claude-code-plugin](https://github.com/traceroot-ai/traceroot-claude-code-plugin) > > Source and full settings ## Next steps > [card] [Sessions](/docs/tracing/sessions) > > See a whole Claude Code session as one timeline. > [card] [Data Capture and Privacy](/docs/tracing/data-capture) > > Control what leaves your machine. # Codex Source: https://traceroot.ai/docs/integrations/codex Trace OpenAI Codex sessions with the traceroot plugin Trace every Codex turn — model calls, tool calls, sub-agents and token usage — live, with the [traceroot Codex plugin](https://github.com/traceroot-ai/traceroot-codex-plugin). ## Install Requires Node.js 22 or later. ```bash theme={null} codex plugin marketplace add traceroot-ai/traceroot-codex-plugin codex plugin add tracing@traceroot-codex-plugin codex features enable hooks ``` Start `codex` and choose **Trust all and continue** at the "Hooks need review" prompt, or trust them later with `/hooks`. ## Set up Create `~/.codex/traceroot.json`: ```json theme={null} { "enabled": true, "api_key": "your-key-from-app.traceroot.ai" } ``` The plugin runs as a Codex hook, which doesn't reliably inherit your shell environment, so the config file is preferred. | Setting | Env var | Default | Description | | - | - | - | - | | `enabled` | `TRACE_TO_TRACEROOT` | `false` | Turns tracing on. An API key must also be set. | | `api_key` | `TRACEROOT_CODEX_API_KEY` or `TRACEROOT_API_KEY` | — | Required. | | `host_url` | `TRACEROOT_CODEX_HOST_URL` or `TRACEROOT_HOST_URL` | `https://app.traceroot.ai` | Set only when self-hosting. | | `user_id` | `TRACEROOT_CODEX_USER_ID` | — | Recorded on each trace. | | `capture_images` | `TRACEROOT_CODEX_CAPTURE_IMAGES` | `false` | Send screenshots from tool output and images attached to prompts. | | `debug` | `TRACEROOT_CODEX_DEBUG` | `false` | Log diagnostics to stderr. | Environment variables win over a project `.codex/traceroot.json`, which wins over the global file. `api_key`, `host_url` and `capture_images` are read only from environment variables or the global file, so a repository you open can't redirect your traces. The [plugin README](https://github.com/traceroot-ai/traceroot-codex-plugin#configure) lists every setting. ## Usage Use Codex as usual. Each turn becomes one trace, sent live while Codex works. Traces from the same thread share a session ID. ``` Codex Turn # AGENT span — one per turn ├── gpt-… # LLM span — tokens including reasoning │ ├── exec_command # TOOL span — command, exit code, output │ ├── exec │ └── spawn_agent # TOOL span — a sub-agent's turn ``` For `codex exec` and other runs without a prompt, trust has to be in place first, because untrusted hooks are skipped without any message. Run `codex exec --skip-git-repo-check "echo hello" < /dev/null`; the output should include `hook: Stop Completed`. The [README](https://github.com/traceroot-ai/traceroot-codex-plugin#headless-runs) shows how to carry trust to CI. ## Verify Run one prompt in Codex and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `Codex Turn`. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Turns | One trace per turn, grouped by Codex thread | | Model calls | Model id, prompt and response | | Tokens | Input, output and reasoning counts | | Tool calls | Tool name, input, output, status and exit code | | Sub-agents | Sub-agent turns under the tool span that started them | | Images | Screenshots and prompt images, when `capture_images` is on | | User, environment, git | `user_id`, `environment`, repository and commit | Prompts, tool inputs and tool outputs are sent as they are, including commands and file contents. Nothing is redacted. Self-hosted instances store images as attachments when `ATTACHMENT_ENABLED=true`. See [Data Capture and Privacy](/docs/tracing/data-capture#what-is-captured-by-default). ## Limitations * If no trace arrives, check that the hooks are trusted (`/hooks`) and that `enabled` is `true`. Set `debug` to `true` (`TRACEROOT_CODEX_DEBUG`) to log to stderr. * Codex asks you to trust the hooks again after every plugin update. Until you do, no traces are sent. ## Run the example > [card] [traceroot-codex-plugin](https://github.com/traceroot-ai/traceroot-codex-plugin) > > Source, full settings and headless setup ## Next steps > [card] [Sessions](/docs/tracing/sessions) > > See a whole Codex thread as one timeline. > [card] [Data Capture and Privacy](/docs/tracing/data-capture) > > Control what leaves your machine. # CrewAI Source: https://traceroot.ai/docs/integrations/crewai Auto-instrument CrewAI multi-agent collaborative workflows Automatically capture agent roles, tasks, crew executions, and tool invocations within the CrewAI framework. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot crewai > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.CREWAI, Integration.OPENAI]) > ``` ## Usage Once initialized, crew and agent executions are captured automatically: ```python theme={null} from crewai import Agent, Task, Crew researcher = Agent( role="Research Analyst", goal="Uncover the latest developments in AI agent frameworks", backstory="You are an expert at identifying emerging trends in AI.", ) writer = Agent( role="Tech Writer", goal="Craft a compelling summary of AI agent framework trends", backstory="You turn complex technical insights into engaging narratives.", ) research_task = Task( description="Analyze the latest AI agent frameworks and summarize key trends.", expected_output="A concise research summary", agent=researcher, ) write_task = Task( description="Write a short article based on the research summary.", expected_output="A short article on AI agent framework trends", agent=writer, context=[research_task], ) crew = Crew(agents=[researcher, writer], tasks=[research_task, write_task]) # The entire crew execution is automatically traced result = crew.kickoff() ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `Crew_.kickoff`; naming the crew with `Crew(name=...)` uses that name instead. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Crew execution | The overarching `kickoff()` session | | Agent roles | Each agent's role, goal, and backstory as span metadata | | Task steps | Individual task executions with inputs and outputs | | Tool calls | Tool invocations with input arguments and results | | LLM calls | Raw completion requests to the provider | | Delegation hierarchy | Manager-agent relationships and task assignments | | Token usage | Aggregated across all LLM calls | | Cost | Aggregated when the underlying model IDs match traceroot's pricing table | ## Limitations * CrewAI delegates LLM generation to underlying providers (e.g., OpenAI, Anthropic). Initializing your provider integration (such as `Integration.OPENAI`) alongside CrewAI is recommended for accurate token usage and cost calculation. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/crewai-agent) > > Run the Python example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # DSPy Source: https://traceroot.ai/docs/integrations/dspy Auto-instrument DSPy modules, signatures, and chain-of-thought reasoning Automatically capture module executions, signature predictions, and underlying LLM calls in [DSPy](https://github.com/stanfordnlp/dspy) programs. ## Install > [tab] Python > > ```bash theme={null} > pip install "traceroot[dspy]" > ``` The `dspy` extra brings in DSPy itself along with the OpenInference instrumentor that emits its spans. ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.DSPY]) > ``` ## Usage Once initialized, every DSPy module call and underlying LLM request is traced automatically: ```python theme={null} import dspy import traceroot from traceroot import Integration traceroot.initialize(integrations=[Integration.DSPY]) # DSPy resolves the API key from OPENAI_API_KEY in the environment. dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", max_tokens=1024)) class CoTQA(dspy.Module): """Chain-of-thought question-answering module.""" def __init__(self): super().__init__() self.cot = dspy.ChainOfThought("question -> answer") def forward(self, question: str): return self.cot(question=question) qa = CoTQA() # The forward call, the chain-of-thought step, and the LLM call are all captured result = qa(question="Why does ice float on water?") print(result.reasoning) print(result.answer) traceroot.flush() ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `CoTQA.forward` — the outer module — with `ChainOfThought.forward` and `Predict(StringSignature).forward` nested beneath it. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Module calls | Each `Module.__call__` / `Module.forward` invocation | | Predictors | `Predict`, `ChainOfThought`, `ReAct`, etc. as nested spans | | Signatures | Input/output fields declared on each signature | | LLM calls | Raw completion requests to the configured `dspy.LM` | | Tokens & Cost | Aggregated token usage; cost when the model ID matches traceroot's pricing table | | Latency | Duration per module call and per LLM call | ## Limitations * Requires installing the `dspy` extra via `pip install "traceroot[dspy]"`. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/dspy-agent) > > Run the Python example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # Exa Source: https://traceroot.ai/docs/integrations/exa Record the cost of Exa searches on their tool spans Record what each [Exa](https://exa.ai) search costs next to your LLM spend. Every search becomes a tool span with its query, results, latency and dollar cost, and the cost counts toward the trace total. > [!NOTE] > > traceroot does not auto-instrument Exa. You trace the search with `observe` and report its cost yourself, and traceroot stores the cost you report. See [Tool and API calls](/docs/tracing/cost-tracking#tool-and-api-calls) for how reported costs add up. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot exa-py > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot exa-js > ``` Reporting a cost requires `traceroot` 0.5.0 or later, or `@traceroot-ai/traceroot` 0.7.0 or later. ## Set up Initialize traceroot once at startup. No integration flag is needed: the `observe` wrapper in the next step traces the search. > [tab] Python > > ```python theme={null} > import traceroot > > traceroot.initialize() > ``` > [tab] TypeScript > > ```typescript theme={null} > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize(); > ``` ## Usage Exa returns the dollar cost of each request in `costDollars`. To record it: 1. Wrap the search in a function traced as a tool span (`type="tool"`). 2. Read `costDollars.total` from the response. 3. Report it with `update_current_span(cost=...)` (`updateCurrentSpan` in TypeScript) while that span is open. > [tab] Python > > ```python theme={null} > from exa_py import Exa > from traceroot import observe, update_current_span > > exa = Exa() # reads EXA_API_KEY > > @observe(name="exa_search", type="tool") > def web_search(query: str): > response = exa.search(query, num_results=3) > if response.cost_dollars: > update_current_span(cost=response.cost_dollars.total) > return response > ``` > [tab] TypeScript > > ```typescript theme={null} > import Exa from 'exa-js'; > import { observe, updateCurrentSpan } from '@traceroot-ai/traceroot'; > > const exa = new Exa(process.env.EXA_API_KEY); > > async function webSearch(query: string) { > return observe({ name: 'exa_search', type: 'tool' }, async (q: string) => { > const response = await exa.search(q, { numResults: 3 }); > if (response.costDollars) updateCurrentSpan({ cost: response.costDollars.total }); > return response; > }, query); > } > ``` Call `web_search` from your agent like any other function. Its span nests under the span that called it, and its cost adds to that trace's total. ![Span details for an exa_search tool span with a cost of 0.007000, its search query input and the search results output](https://mintcdn.com/tracerootai/3SDwRl5xRxfZvr-4/images/tool_cost_price_in_response_v1.png?fit=max\&auto=format\&n=3SDwRl5xRxfZvr-4\&q=85\&s=6863aef569ace737aaa4d86c9d313ff5) The tool span carries the price the search API returned ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see an `exa_search` span, at the root or under your agent's span. Open it to see its cost; the trace's total cost includes it. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Input | The query passed to `web_search` | | Output | The search response | | Cost | The `costDollars.total` you report, in USD | | Latency | Request duration | ## Limitations * A response without `costDollars` records no cost on the span. ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Price in the response](/docs/tracing/cost-tracking#price-in-the-response) > > Report the price a provider returns with each request # Google Gemini Source: https://traceroot.ai/docs/integrations/gemini Auto-instrument Google Gemini API calls via the GenAI SDK Automatically capture all Gemini model calls made through the `google-genai` SDK. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot google-genai > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.GOOGLE_GENAI]) > ``` ## Usage Once initialized, all Gemini calls are captured automatically — including tool-call requests: ```python theme={null} import os from google import genai from google.genai import types client = genai.Client(api_key=os.environ["GEMINI_API_KEY"]) weather_tool = types.Tool(function_declarations=[ types.FunctionDeclaration( name="get_weather", description="Get current weather for a city", parameters=types.Schema( type="OBJECT", properties={"city": types.Schema(type="STRING", description="City name")}, required=["city"], ), ) ]) # This call is automatically traced response = client.models.generate_content( model="gemini-3.8-flash", contents="What's the weather in Tokyo?", config=types.GenerateContentConfig(tools=[weather_tool]), ) # The model asks to call get_weather; run it and send the result back yourself print(response.function_calls) ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `GenerateContent` (or `GenerateContentStream` when streaming). 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Model | `gemini-3.8-flash`, `gemini-3.5-flash-lite`, `gemini-3.1-pro-preview`, etc. | | Contents | Input messages | | Response | Generated text and function calls | | Tokens | Input and output token counts | | Cost | Calculated when the model ID matches traceroot's pricing table | | Latency | Request duration | ## Limitations * When passing Python functions as `tools`, the SDK's automatic function calling executes the entire tool loop inside a single `generate_content` call, which produces one span priced on the final model call only. For accurate per-step spans and cost, pass function declarations or set `automatic_function_calling=types.AutomaticFunctionCallingConfig(disable=True)` and run the loop manually. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/gemini-tool-agent) > > Run the Python example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # Google ADK Source: https://traceroot.ai/docs/integrations/google-adk Auto-instrument Google Agent Development Kit agents, workflows, tools, and runners Automatically capture agent runs, workflow graphs, tool executions, and the underlying Gemini calls within the [Google Agent Development Kit (ADK)](https://github.com/google/adk-python). ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot google-adk > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.GOOGLE_ADK]) > ``` ## Usage Once initialized, ADK agent runs, tool calls, and Gemini calls are captured automatically: ```python theme={null} import asyncio import traceroot from traceroot import Integration from google.adk import Agent from google.adk.runners import InMemoryRunner from google.genai import types traceroot.initialize(integrations=[Integration.GOOGLE_ADK]) def get_weather(city: str) -> dict: """Get current weather for a city.""" return {"status": "success", "city": city, "temp": 72, "condition": "sunny"} # ADK resolves the Gemini API key from GOOGLE_API_KEY in the environment. agent = Agent( name="assistant", model="gemini-3.8-flash", description="Helpful assistant with access to tools.", instruction="You are a helpful AI assistant. Use tools when needed.", tools=[get_weather], ) async def main(): runner = InMemoryRunner(agent=agent, app_name="traceroot-adk-demo") await runner.session_service.create_session( app_name="traceroot-adk-demo", user_id="demo-user", session_id="demo-session", ) # The full run, including tool calls and Gemini calls, is traced async for event in runner.run_async( user_id="demo-user", session_id="demo-session", new_message=types.Content( role="user", parts=[types.Part(text="What's the weather in Tokyo?")], ), ): if event.is_final_response() and event.content and event.content.parts: print(event.content.parts[0].text) if __name__ == "__main__": asyncio.run(main()) traceroot.flush() ``` ### Usage with the ADK CLI If you run your agent with `adk run` or `adk web`, initialize traceroot at the top of `agent.py`: ```python theme={null} # my_agent/agent.py import traceroot from traceroot import Integration from google.adk import Agent traceroot.initialize(integrations=[Integration.GOOGLE_ADK]) def get_current_time(city: str) -> dict: """Returns the current time in a specified city.""" return {"status": "success", "city": city, "time": "10:30 AM"} root_agent = Agent( model="gemini-3.8-flash", name="root_agent", description="Tells the current time in a specified city.", instruction="Use the 'get_current_time' tool to answer.", tools=[get_current_time], ) ``` > [!NOTE] > > `adk web` and `adk api_server` install their own OpenTelemetry tracer provider before `agent.py` loads. From `traceroot` 0.3.0, `@observe` spans are traced alongside ADK's spans in these modes. On 0.2.0, ADK's spans still arrive but `@observe` spans are dropped; use `adk run` or your own runner instead. ### Usage with Workflows ADK 2.x runs agents, tools, and plain Python functions as nodes of a graph `Workflow`. Each workflow run, node, fan-out, and join is traced: ```python theme={null} import asyncio import traceroot from traceroot import Integration from google.adk import Agent, Event, Workflow from google.adk.runners import InMemoryRunner from google.adk.workflow import JoinNode from google.genai import types traceroot.initialize(integrations=[Integration.GOOGLE_ADK]) def record_city(node_input: str): """Store the city in session state and pass it on to both branches.""" city = node_input.strip() return Event(output=city, state={"city": city}) def get_weather(city: str) -> dict: """Get current weather for the city in session state.""" return {"city": city, "temp": 72, "condition": "sunny"} suggest_food = Agent( name="suggest_food", model="gemini-3.8-flash", instruction="In one sentence, name a signature local dish to try in {city}.", ) # START -> record_city -> (get_weather, suggest_food in parallel) -> gather workflow = Workflow( name="city_brief", edges=[("START", record_city, (get_weather, suggest_food), JoinNode(name="gather"))], ) async def main(): runner = InMemoryRunner(agent=workflow, app_name="traceroot-adk-workflow") await runner.session_service.create_session( app_name="traceroot-adk-workflow", user_id="demo-user", session_id="demo-session", ) # The workflow run and every node in it are traced output = None async for event in runner.run_async( user_id="demo-user", session_id="demo-session", new_message=types.Content(role="user", parts=[types.Part(text="Tokyo")]), ): if event.output is not None: output = event.output # The last output is the joined result, keyed by branch name print(output) if __name__ == "__main__": asyncio.run(main()) traceroot.flush() ``` > [!NOTE] > > Workflow and node spans need the first `traceroot` release after 0.2.0. On 0.2.0, only the `Agent` nodes of a workflow appear. ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `invocation []`, or `invoke_workflow ` for workflow runs (requires a `traceroot` release newer than 0.2.0). 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Runner calls | An `invocation []` span per runner call | | Workflow runs | An `invoke_workflow ` span per workflow run | | Workflow nodes | An `invoke_node ` span per function or join node; `Agent` nodes appear as agent runs | | Agent runs | An `agent_run []` agent span per agent and sub-agent run | | LLM calls | A `call_llm` span per Gemini call with model, request, and response | | Tool calls | A tool span per tool call, named after the tool, with input arguments and result | | Tokens & Cost | Input, output, and reasoning token counts per LLM call; cost for models in the pricing table, including `gemini-3.8-flash` | | Sessions & users | The ADK session id as `session.id` and user id as `user.id` on every span | | Latency | Duration per agent run and per span | ## Limitations * Model aliases such as `gemini-flash-latest` are not priced; pin a concrete model ID (such as `gemini-3.8-flash`) to capture token costs. * `adk web` and `adk api_server` install their own OpenTelemetry tracer provider before `agent.py` loads; traceroot >= 0.3.0 is recommended when using these modes. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Agent](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/google-adk-agent) > > Run the Python agent example > [card] [Workflow](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/google-adk-workflow) > > Run the Python workflow example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # Supported Models Source: https://traceroot.ai/docs/integrations/internal-models All LLM models available natively in traceroot Cloud traceroot Cloud provides the following models out of the box — no API key required. ## Anthropic | Model | Description | | - | - | | `claude-opus-4-8` | Most capable model for complex reasoning and debugging | | `claude-opus-4-7` | Previous Opus generation | | `claude-opus-4-6` | Earlier Opus generation | | `claude-sonnet-5` | Sonnet 5 for coding and agentic tasks | | `claude-sonnet-5-5` | Adaptive thinking with a 1M-token context window | | `claude-sonnet-4-6` | Balanced performance and speed | | `claude-opus-4-5` | Earlier Opus generation | | `claude-sonnet-4-5` | Previous Sonnet generation | | `claude-haiku-5-5` | Fast, low-cost; detector fallback without OpenAI key | | `claude-haiku-4-5` | Fast and lightweight | ## OpenAI | Model | Description | | - | - | | `gpt-5.5` | Default OpenAI model | | `gpt-6-luna` | Fast, low-cost GPT-6 model; the detector default | | `gpt-5.4` | Balanced GPT-5.4 model | | `gpt-5.4-mini` | Fast and cost-efficient GPT-5.4 variant | | `gpt-5.4-nano` | Smallest, lowest-cost GPT-5.4 variant | | `gpt-5` | Previous flagship model | | `gpt-5-mini` | Fast and cost-efficient GPT-5 variant | | `o3` | Advanced reasoning model | | `gpt-5.3-codex` | Code-optimized model via Responses API | | `gpt-5.6-sol` | GPT-5.6 for complex reasoning and tool use | | `gpt-5.6-terra` | Cost-efficient GPT-5.6 reasoning model | | `gpt-6-sol` | GPT-6 for reasoning, coding, and tool use | ## Bring Your Own Key (BYOK) Need a model not listed here? With [BYOK](/docs/ai-agent/byok) you can bring your own API key and use any supported provider — including Azure OpenAI, Google (Gemini), Amazon Bedrock, xAI, DeepSeek, OpenRouter, Kimi, GLM, and TypeSafe AI (detectors only). > [card] [Set up BYOK](/docs/ai-agent/byok) > > Use your own API keys to control which model powers the traceroot Agent. # LangChain & LangGraph Source: https://traceroot.ai/docs/integrations/langchain Auto-instrument LangChain and LangGraph agent frameworks Automatically capture agent steps, tool calls, and LLM invocations within LangChain chains and LangGraph graphs. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot langchain-core langchain-openai langgraph > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot @langchain/core @langchain/openai @langchain/langgraph > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.LANGCHAIN]) > ``` > [tab] TypeScript > > ```typescript theme={null} > import * as lcCallbackManager from '@langchain/core/callbacks/manager'; > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize({ > instrumentModules: { langchain: lcCallbackManager }, > }); > ``` ## Usage ### Usage with LangGraph > [tab] Python > > ```python theme={null} > from langchain_openai import ChatOpenAI > from langgraph.prebuilt import create_react_agent > > llm = ChatOpenAI(model="gpt-4o") > > tools = [search_tool, calculator_tool] > agent = create_react_agent(llm, tools=tools) > > # All agent steps are automatically traced > result = agent.invoke({ > "messages": [{"role": "user", "content": "What is 2 + 2?"}] > }) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { ChatOpenAI } from '@langchain/openai'; > import { HumanMessage } from '@langchain/core/messages'; > import { END, START, StateGraph, Annotation } from '@langchain/langgraph'; > > const llm = new ChatOpenAI({ model: 'gpt-4o', temperature: 0 }); > > const AgentState = Annotation.Root({ > query: Annotation({ reducer: (_, v) => v, default: () => '' }), > answer: Annotation({ reducer: (_, v) => v, default: () => '' }), > }); > > // Build graph — all nodes are automatically traced > const app = new StateGraph(AgentState) > .addNode('answer', async (state) => { > const response = await llm.invoke([new HumanMessage(state.query)]); > return { answer: response.content as string }; > }) > .addEdge(START, 'answer') > .addEdge('answer', END) > .compile(); > > const result = await app.invoke({ query: 'What is 2 + 2?' }); > console.log(result.answer); > ``` ### Usage with LangChain > [tab] Python > > ```python theme={null} > from langchain_openai import ChatOpenAI > from langchain_core.prompts import ChatPromptTemplate > > llm = ChatOpenAI(model="gpt-4o") > prompt = ChatPromptTemplate.from_messages([ > ("system", "You are a helpful assistant."), > ("user", "{input}"), > ]) > > chain = prompt | llm > > # Chain execution is automatically traced > result = chain.invoke({"input": "Hello!"}) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { ChatOpenAI } from '@langchain/openai'; > import { ChatPromptTemplate } from '@langchain/core/prompts'; > > const llm = new ChatOpenAI({ model: 'gpt-4o' }); > const prompt = ChatPromptTemplate.fromMessages([ > ['system', 'You are a helpful assistant.'], > ['user', '{input}'], > ]); > > const chain = prompt.pipe(llm); > > // Chain execution is automatically traced > const result = await chain.invoke({ input: 'Hello!' }); > ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `RunnableSequence` or `CompiledStateGraph`. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Agent steps | Each agent iteration as a span | | Tool calls | Each tool invocation with input/output | | LLM calls | All LLM calls within the chain/graph | | Graph structure | Parent-child relationships between steps | | Token usage | Aggregated across all LLM calls | | Cost | Aggregated when the underlying model IDs match traceroot's pricing table | ## Limitations * For TypeScript, `instrumentModules` must be configured with `@langchain/core/callbacks/manager` to enable automatic callback tracing. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/langgraph-code-agent) > > Run the Python example > [card] [TypeScript](https://github.com/traceroot-ai/traceroot/tree/main/examples/typescript/langchain) > > Run the TypeScript example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [TypeScript SDK Reference](/docs/tracing/typescript-sdk) > > Explore the full TypeScript SDK reference and configuration options. # LangChain DeepAgents Source: https://traceroot.ai/docs/integrations/langchain-deepagents Trace multi-agent research pipelines built with DeepAgents Automatically capture the full multi-agent hierarchy of [DeepAgents](https://github.com/langchain-ai/deepagentsjs) pipelines. DeepAgents builds on LangChain, so traceroot instruments it via the LangChain callback manager. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot deepagents langchain-core langchain-anthropic > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot deepagents @langchain/core @langchain/anthropic > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.LANGCHAIN]) > ``` > [tab] TypeScript > > ```typescript theme={null} > import * as lcCallbackManager from '@langchain/core/callbacks/manager'; > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize({ > instrumentModules: { langchain: lcCallbackManager }, > }); > ``` ## Usage > [tab] Python > > ```python theme={null} > from traceroot import observe, using_attributes > from deepagents import create_deep_agent > > research_subagent = { > "name": "research-agent", > "description": "Searches the web to gather information on a topic.", > "system_prompt": "You are a thorough research agent. Cite sources and organise findings clearly.", > "tools": [web_search], > } > > supervisor = create_deep_agent( > model="claude-sonnet-4-6", > system_prompt="You are a research supervisor. Delegate tasks to sub-agents and synthesise their outputs.", > subagents=[research_subagent], > ) > > @observe(name="research_session", type="agent") > def run_research(query: str) -> str: > result = supervisor.invoke({"messages": [{"role": "user", "content": query}]}) > messages = result.get("messages", []) > return messages[-1].content if messages else "" > > with using_attributes(user_id="demo-user", session_id="research-session"): > report = run_research("What are the latest AI agent frameworks in 2025?") > ``` > [tab] TypeScript > > ```typescript theme={null} > import { observe, usingAttributes } from '@traceroot-ai/traceroot'; > import { createDeepAgent, type SubAgent } from 'deepagents'; > import { ChatAnthropic } from '@langchain/anthropic'; > import { HumanMessage } from '@langchain/core/messages'; > > const llm = new ChatAnthropic({ model: 'claude-sonnet-4-20250514', temperature: 0 }); > > const researchSubAgent: SubAgent = { > name: 'research-agent', > description: 'Researches a specific topic in depth.', > systemPrompt: 'You are a thorough research agent. Gather comprehensive, current information and cite sources.', > tools: [internetSearch], > }; > > const supervisor = createDeepAgent({ > model: llm, > tools: [internetSearch], > systemPrompt: 'You are a research supervisor. Delegate tasks to sub-agents and synthesise their outputs.', > subagents: [researchSubAgent], > }); > > await usingAttributes( > { userId: 'demo-user', sessionId: 'research-session' }, > () => > observe({ name: 'research_session', type: 'agent' }, async () => { > const result = await supervisor.invoke( > { messages: [new HumanMessage('What are the latest AI agent frameworks in 2025?')] }, > { recursionLimit: 100 }, > ); > const messages = result.messages ?? []; > console.log(messages[messages.length - 1]?.content); > }), > ); > ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `research_session`, the name passed to `observe`. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Supervisor steps | Each supervisor reasoning and delegation step | | Sub-agent calls | Each sub-agent invocation with its own span | | Tool calls | Tool inputs and outputs within sub-agents | | LLM calls | All LLM calls across the full agent hierarchy | | Token usage | Aggregated across all agents and LLM calls | | Cost | Aggregated when the underlying model IDs match traceroot's pricing table | ## Limitations * DeepAgents is built on LangChain. For TypeScript projects, `instrumentModules` must be configured with `@langchain/core/callbacks/manager`. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/deepagents) > > Run the Python example > [card] [TypeScript](https://github.com/traceroot-ai/traceroot/tree/main/examples/typescript/deepagents) > > Run the TypeScript example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [TypeScript SDK Reference](/docs/tracing/typescript-sdk) > > Explore the full TypeScript SDK reference and configuration options. # LlamaIndex Source: https://traceroot.ai/docs/integrations/llamaindex Auto-instrument LlamaIndex RAG pipelines and query engines Automatically capture document ingestion, embedding, retrieval, and LLM synthesis within LlamaIndex pipelines. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot llama-index-core llama-index-llms-openai llama-index-embeddings-openai > ``` ## Set up To capture the complete picture—including both the pipeline structure and the underlying token costs—we highly recommend initializing both the LlamaIndex integration and your specific LLM provider (e.g., OpenAI, Anthropic, Google GenAI). > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > # Initialize LlamaIndex alongside your LLM provider to capture tokens and costs > traceroot.initialize(integrations=[ > Integration.LLAMA_INDEX, > Integration.OPENAI # Or ANTHROPIC, GOOGLE_GENAI, etc. > ]) > ``` ## Usage Once initialized, all LlamaIndex operations are captured automatically: ```python theme={null} from llama_index.core import Document, Settings, VectorStoreIndex from llama_index.llms.openai import OpenAI from llama_index.embeddings.openai import OpenAIEmbedding Settings.llm = OpenAI(model="gpt-4o-mini") Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small") documents = [ Document(text="traceroot is an open-source AI agent improvement platform."), Document(text="traceroot supports OpenTelemetry-compatible tracing via a Python SDK."), ] # Index construction is automatically traced (SentenceSplitter, Embedding) index = VectorStoreIndex.from_documents(documents) query_engine = index.as_query_engine() # Each query is automatically traced (retrieval + LLM synthesis) response = query_engine.query("What is traceroot?") print(response) ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `RetrieverQueryEngine.query`, the class and method that `as_query_engine().query()` resolves to. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Document chunking | Each `SentenceSplitter` call as a span | | Embedding calls | Each batch embedding with text inputs and vector outputs | | Retrieval | Top-k retrieval with query and retrieved documents | | LLM synthesis | Response generation with context and final answer | | Tokens & Cost | Input/output tokens; cost when the model ID matches traceroot's pricing table | | Latency | Duration of each pipeline stage | ## Limitations * Initializing both `Integration.LLAMA_INDEX` and your underlying LLM provider integration (such as `Integration.OPENAI`) is recommended to capture full token usage and pricing. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/llamaindex-rag) > > Run the Python example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # Mastra Source: https://traceroot.ai/docs/integrations/mastra Export traces from Mastra agents to traceroot via the OTLP exporter Export traces from [Mastra](https://mastra.ai) agents to traceroot using the `@traceroot-ai/mastra` exporter package. This integrates with Mastra's built-in OpenTelemetry observability system. ## Install > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/mastra @mastra/core @mastra/observability > ``` ## Set up Configure traceroot as an exporter in your Mastra instance: ```typescript theme={null} import { Mastra } from '@mastra/core'; import { Observability } from '@mastra/observability'; import { TraceRootExporter } from '@traceroot-ai/mastra'; const exporter = new TraceRootExporter({ apiKey: process.env.TRACEROOT_API_KEY, }); const mastra = new Mastra({ agents: { /* your agents */ }, observability: new Observability({ configs: { traceroot: { serviceName: 'my-mastra-app', exporters: [exporter], }, }, }), }); ``` ## Usage Once configured, all agent calls are traced automatically: ```typescript theme={null} import { Agent } from '@mastra/core/agent'; import { anthropic } from '@ai-sdk/anthropic'; const weatherAgent = new Agent({ id: 'weatherAgent', name: 'Weather Agent', instructions: 'You are a helpful weather assistant.', model: anthropic('claude-haiku-4-5-20251001'), tools: { /* your tools */ }, }); const mastra = new Mastra({ agents: { weatherAgent }, observability: new Observability({ configs: { traceroot: { serviceName: 'mastra-weather-agent', exporters: [exporter], }, }, }), }); const agent = mastra.getAgent('weatherAgent'); // Pass a consistent threadId to group calls under the same session const result = await agent.generate("What's the weather in Tokyo?", { threadId: 'session-123', resourceId: 'user-456', }); console.log(result.text); // Flush traces before process exit await exporter.flush(); ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `agent run: 'weatherAgent'`, named after the agent id. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Agent name | The Mastra agent ID | | Model | Underlying LLM model name | | Messages | Input messages and conversation history | | Response | Generated text output | | Tool calls | Each tool invocation with input and output | | Tokens | Input and output token counts | | Cost | Calculated when the model ID matches traceroot's pricing table | | Latency | Request duration per span | ## Limitations * Tracing requires exporting spans through the `@traceroot-ai/mastra` exporter package configured in Mastra's `Observability` system. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [TypeScript](https://github.com/traceroot-ai/traceroot/tree/main/examples/typescript/mastra) > > Run the TypeScript example ## Next steps > [card] [TypeScript SDK Reference](/docs/tracing/typescript-sdk) > > Explore the full TypeScript SDK reference and configuration options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # Microsoft Agent Framework Source: https://traceroot.ai/docs/integrations/microsoft-agent-framework Auto-instrument Microsoft Agent Framework agent runs, model calls, and tool executions Automatically capture agent runs, model calls, and tool executions from [Microsoft Agent Framework](https://github.com/microsoft/agent-framework) through its built-in OpenTelemetry emission — no manual instrumentation of the framework required. ## Install > [tab] Python > > ```bash theme={null} > pip install "traceroot[agent-framework]" agent-framework-openai > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.AGENT_FRAMEWORK]) > ``` Initialize traceroot **before** importing `agent_framework`, so the OpenTelemetry instrumentation is wired up before the framework emits its first span. ## Usage Once initialized, every agent invocation, model call, and tool execution is captured automatically: ```python theme={null} import asyncio import traceroot from traceroot import Integration traceroot.initialize(integrations=[Integration.AGENT_FRAMEWORK]) # Import after initialize() so tracing is already active. from agent_framework import Agent from agent_framework.openai import OpenAIChatClient def get_weather(city: str) -> str: """Return a short weather forecast for a city.""" return f"{city}: 18°C, partly cloudy." agent = Agent( OpenAIChatClient(model="gpt-4o-mini"), "You are a helpful travel concierge. Use the weather tool when relevant.", name="Concierge", tools=[get_weather], ) async def main(): # The agent run, the model call, and the get_weather tool call are all traced. response = await agent.run("What's the weather in Tokyo?") print(response.text) asyncio.run(main()) ``` Tools are plain functions — Agent Framework already emits an `execute_tool` span for each call, so they don't need manual decoration. ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `invoke_agent Concierge` — Agent Framework names each agent span `invoke_agent `. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Agent run | Each agent invocation as a span (nested for delegated agents) | | Model | The chat model used for each call | | Tool calls | An `execute_tool` span per tool execution, with arguments | | Response | Generated text and tool outputs | | Tokens | Input and output token counts | | Cost | Calculated when the model ID matches traceroot's pricing table | | Latency | Request duration | ## Limitations * Requires installing the `agent-framework` extra via `pip install "traceroot[agent-framework]"`. * traceroot must be initialized before importing `agent_framework` so that OpenTelemetry listeners are attached before spans are generated. Requires OpenTelemetry >= 1.39. ## Run the example Clone the repo and run a complete multi-agent example end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/microsoft-agent-framework) > > Run the Python example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # Mistral Source: https://traceroot.ai/docs/integrations/mistral Auto-instrument Mistral AI API calls Automatically capture all Mistral AI chat completion calls, including tool/function calls and streaming responses. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot mistralai > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.MISTRAL]) > ``` ## Usage Once initialized, all Mistral calls are captured automatically: ```python theme={null} import os from mistralai.client import Mistral import traceroot from traceroot import Integration traceroot.initialize(integrations=[Integration.MISTRAL]) client = Mistral(api_key=os.environ["MISTRAL_API_KEY"]) # This call is automatically traced response = client.chat.complete( model="mistral-large-latest", messages=[ {"role": "user", "content": "What is the capital of France?"}, ], ) print(response.choices[0].message.content) ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `Chat.complete` (or `Chat.stream` when streaming). 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Model | `mistral-large-latest`, `mistral-small-latest`, etc. | | Messages | Input messages array | | Response | Completion content | | Tool calls | Function calls emitted by the model | | Tokens | Input and output tokens | | Cost | Not currently calculated for Mistral model IDs; token usage is still captured | | Latency | Request duration | ## Limitations None known. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/mistral-tool-agent) > > Run the Python example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # OpenAI Source: https://traceroot.ai/docs/integrations/openai Auto-instrument OpenAI API calls Automatically capture all OpenAI Chat Completions and Responses API calls. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot openai > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot openai > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.OPENAI]) > ``` > [tab] TypeScript > > ```typescript theme={null} > import OpenAI from 'openai'; > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize({ > instrumentModules: { openAI: OpenAI }, > }); > > const openai = new OpenAI(); > ``` ## Usage > [tab] Python > > Once initialized, all OpenAI calls are captured automatically: > > ```python theme={null} > import openai > > client = openai.OpenAI() > > # This call is automatically traced > response = client.chat.completions.create( > model="gpt-4o", > messages=[ > {"role": "system", "content": "You are a helpful assistant."}, > {"role": "user", "content": "What is the capital of France?"}, > ], > ) > > print(response.choices[0].message.content) > ``` > [tab] TypeScript > > ```typescript theme={null} > import OpenAI from 'openai'; > > const openai = new OpenAI(); > > // This call is automatically traced > const response = await openai.chat.completions.create({ > model: 'gpt-4o', > messages: [ > { role: 'system', content: 'You are a helpful assistant.' }, > { role: 'user', content: 'What is the capital of France?' }, > ], > }); > > console.log(response.choices[0].message.content); > ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `ChatCompletion` (`OpenAI Chat Completions` on TypeScript). 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Model | `gpt-4o`, `gpt-4o-mini`, `o3`, etc. | | Messages | Input messages array | | Response | Completion content | | Tokens | Prompt, completion, and total tokens | | Cost | Calculated when the model ID matches traceroot's pricing table | | Latency | Request duration | ## Limitations None known. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/openai-tool-agent) > > Run the Python example > [card] [TypeScript](https://github.com/traceroot-ai/traceroot/tree/main/examples/typescript/openai) > > Run the TypeScript example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [TypeScript SDK Reference](/docs/tracing/typescript-sdk) > > Explore the full TypeScript SDK reference and configuration options. # OpenAI Agents SDK Source: https://traceroot.ai/docs/integrations/openai-agents-sdk Auto-instrument OpenAI Agents SDK agents, tools, and multi-turn conversations Automatically capture agent runs, tool executions, and handoff transitions within the OpenAI Agents SDK ([Python](https://github.com/openai/openai-agents-python), [TypeScript](https://github.com/openai/openai-agents-js)). ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot openai-agents > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot @openai/agents zod > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.OPENAI_AGENTS]) > ``` > [tab] TypeScript > > ```typescript theme={null} > import * as agents from '@openai/agents'; > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize({ > instrumentModules: { openaiAgents: agents }, > }); > ``` ## Usage Once initialized, all agent runs and tool calls are captured automatically: > [tab] Python > > ```python theme={null} > import asyncio > import traceroot > from traceroot import Integration > from agents import Agent, Runner, function_tool > > traceroot.initialize(integrations=[Integration.OPENAI_AGENTS]) > > > @function_tool > def get_weather(city: str) -> str: > """Get current weather for a city.""" > return f"The weather in {city} is 72°F and sunny." > > > agent = Agent( > name="Weather Assistant", > instructions="You are a helpful weather assistant.", > model="gpt-4o-mini", > tools=[get_weather], > ) > > > async def main(): > # The entire agent run is automatically traced > result = await Runner.run(agent, "What's the weather in Tokyo?") > print(result.final_output) > > > if __name__ == "__main__": > asyncio.run(main()) > traceroot.flush() > ``` > [tab] TypeScript > > ```typescript theme={null} > import * as agents from '@openai/agents'; > import { Agent, run, tool } from '@openai/agents'; > import { TraceRoot } from '@traceroot-ai/traceroot'; > import { z } from 'zod'; > > TraceRoot.initialize({ > instrumentModules: { openaiAgents: agents }, > }); > > const getWeather = tool({ > name: 'get_weather', > description: 'Get current weather for a city.', > parameters: z.object({ city: z.string() }), > execute: async ({ city }) => `The weather in ${city} is 72°F and sunny.`, > }); > > const agent = new Agent({ > name: 'Weather Assistant', > instructions: 'You are a helpful weather assistant.', > model: 'gpt-4o-mini', > tools: [getWeather], > }); > > async function main() { > // The entire agent run is automatically traced > const result = await run(agent, "What's the weather in Tokyo?"); > console.log(result.finalOutput); > await TraceRoot.shutdown(); > } > > main().catch(console.error); > ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace whose root span is the agent run's workflow name — `Agent workflow` unless you set your own, and `demo_session` in the shipped examples, which wrap the run with `observe`. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Agent runs | Each `Runner.run()` / `run()` invocation | | Agent steps | Individual turns within the agent loop | | Tool calls | Each tool invocation with name, input, and output | | Handoffs | Multi-agent transitions and handoff decisions | | LLM calls | Raw completion requests to OpenAI | | Tokens & Cost | Aggregated token usage; cost when the model ID matches traceroot's pricing table | | Latency | Duration per agent run and per span | ## Limitations None known. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/openai-agents-sdk) > > Run the Python example > [card] [TypeScript](https://github.com/traceroot-ai/traceroot/tree/main/examples/typescript/openai-agents) > > Run the TypeScript example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [TypeScript SDK Reference](/docs/tracing/typescript-sdk) > > Explore the full TypeScript SDK reference and configuration options. # OpenAI-Compatible Providers Source: https://traceroot.ai/docs/integrations/openai-compatible Trace xAI, Moonshot, Z.AI and any other OpenAI-compatible API through the OpenAI integration Any provider that speaks the OpenAI API can be traced through traceroot's OpenAI integration. Use the standard `openai` SDK and point the client at the provider's base URL — no provider-specific instrumentor or enum is required. ## Base URLs | Provider | Base URL | | - | - | | xAI (Grok) | `https://api.x.ai/v1` | | Moonshot (Kimi) | `https://api.moonshot.ai/v1` | | Z.AI (GLM) | `https://open.bigmodel.cn/api/paas/v4` | Any other OpenAI-compatible endpoint works the same way, including self-hosted servers such as vLLM or a LiteLLM proxy: pass its URL as the base URL. [OpenRouter](/docs/integrations/openrouter) follows the same pattern. ## Setup > [tab] Python > > ```python theme={null} > import os > import openai > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.OPENAI]) > > client = openai.OpenAI( > api_key=os.environ["XAI_API_KEY"], > base_url="https://api.x.ai/v1", > ) > ``` > [tab] TypeScript > > ```typescript theme={null} > import OpenAI from 'openai'; > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize({ > instrumentModules: { openAI: OpenAI }, > }); > > const openai = new OpenAI({ > apiKey: process.env.XAI_API_KEY, > baseURL: 'https://api.x.ai/v1', > }); > ``` ## Usage > [tab] Python > > ```python theme={null} > import os > import openai > > client = openai.OpenAI( > api_key=os.environ["XAI_API_KEY"], > base_url="https://api.x.ai/v1", > ) > > # This xAI call is traced by traceroot's OpenAI integration > response = client.chat.completions.create( > model="grok-4.20", > messages=[ > {"role": "system", "content": "You are a helpful assistant."}, > {"role": "user", "content": "What is the capital of France?"}, > ], > ) > > print(response.choices[0].message.content) > ``` > [tab] TypeScript > > ```typescript theme={null} > import OpenAI from 'openai'; > > const openai = new OpenAI({ > apiKey: process.env.XAI_API_KEY, > baseURL: 'https://api.x.ai/v1', > }); > > // This xAI call is traced by traceroot's OpenAI integration > const response = await openai.chat.completions.create({ > model: 'grok-4.20', > messages: [ > { role: 'system', content: 'You are a helpful assistant.' }, > { role: 'user', content: 'What is the capital of France?' }, > ], > }); > > console.log(response.choices[0].message.content); > ``` To use Moonshot or Z.AI instead, swap the API key and base URL from the table above and pass one of that provider's model ids, such as `kimi-k2.6` or `glm-5.1`. ## What Gets Captured | Attribute | Description | | - | - | | Model | The model id sent to the provider, such as `grok-4.20`, `kimi-k2.6` or `glm-5.1` | | Messages | Input messages array | | Response | Completion content | | Tool calls | Function/tool call inputs and outputs when using the OpenAI SDK tool-calling API | | Tokens | Prompt, completion, and total tokens when returned by the provider | | Latency | Request duration | | Cost | Calculated for model ids in traceroot's pricing table | ## Cost Cost is calculated only for model ids listed in traceroot's pricing table ([`standard-model-prices.json`](https://github.com/traceroot-ai/traceroot/blob/main/frontend/packages/core/src/standard-model-prices.json)). Calls to other models are still traced with their messages, tokens and latency, but without a cost. See [Cost Tracking](/docs/tracing/cost-tracking) for details. # OpenRouter Source: https://traceroot.ai/docs/integrations/openrouter Trace OpenRouter calls through the OpenAI-compatible SDK OpenRouter is OpenAI-compatible: use the standard `openai` SDK with the OpenRouter base URL (`https://openrouter.ai/api/v1`). traceroot captures these calls through the existing OpenAI instrumentation — no new OpenRouter-specific instrumentor or enum is required. The same approach works for xAI, Moonshot, Z.AI and other [OpenAI-compatible providers](/docs/integrations/openai-compatible). ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot openai > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot openai > ``` ## Set up > [tab] Python > > ```python theme={null} > import os > import openai > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.OPENAI]) > > client = openai.OpenAI( > api_key=os.environ["OPENROUTER_API_KEY"], > base_url="https://openrouter.ai/api/v1", > ) > ``` > [tab] TypeScript > > ```typescript theme={null} > import OpenAI from 'openai'; > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize({ > instrumentModules: { openAI: OpenAI }, > }); > > const openai = new OpenAI({ > apiKey: process.env.OPENROUTER_API_KEY, > baseURL: 'https://openrouter.ai/api/v1', > }); > ``` ## Usage > [tab] Python > > ```python theme={null} > import os > import openai > > client = openai.OpenAI( > api_key=os.environ["OPENROUTER_API_KEY"], > base_url="https://openrouter.ai/api/v1", > ) > > # This OpenRouter call is traced by traceroot's OpenAI integration > response = client.chat.completions.create( > model="openai/gpt-4o-mini", > messages=[ > {"role": "system", "content": "You are a helpful assistant."}, > {"role": "user", "content": "What is the capital of France?"}, > ], > ) > > print(response.choices[0].message.content) > ``` > [tab] TypeScript > > ```typescript theme={null} > import OpenAI from 'openai'; > > const openai = new OpenAI({ > apiKey: process.env.OPENROUTER_API_KEY, > baseURL: 'https://openrouter.ai/api/v1', > }); > > // This OpenRouter call is traced by traceroot's OpenAI integration > const response = await openai.chat.completions.create({ > model: 'openai/gpt-4o-mini', > messages: [ > { role: 'system', content: 'You are a helpful assistant.' }, > { role: 'user', content: 'What is the capital of France?' }, > ], > }); > > console.log(response.choices[0].message.content); > ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `ChatCompletion` (`OpenAI Chat Completions` on TypeScript). 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Model | Any OpenRouter model string, such as `openai/gpt-4o-mini`, `anthropic/claude-sonnet-4`, or `meta-llama/llama-4-maverick` | | Messages | Input messages array | | Response | Completion content | | Tool calls | Function/tool call inputs and outputs when using the OpenAI SDK tool-calling API | | Tokens | Prompt, completion, and total tokens when returned by the provider | | Cost | Calculated when the routed model ID matches traceroot's pricing table (for example, supported `openai/...`, `anthropic/...`, or `google/...` IDs); otherwise tokens are captured without cost | | Latency | Request duration | ## Limitations * OpenRouter requests are traced through the standard OpenAI integration; token pricing requires provider-returned usage headers or recognized model mappings. ## Run the example Clone the repo and run a complete OpenRouter agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/openrouter-tool-agent) > > Run the Python OpenRouter tool-agent example > [card] [TypeScript](https://github.com/traceroot-ai/traceroot/tree/main/examples/typescript/openrouter) > > Run the TypeScript OpenRouter example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # LLM provider and framework integrations Source: https://traceroot.ai/docs/integrations/overview Find setup guides for OpenAI, LangChain, Claude Code, and other tools in your stack. traceroot integrates with popular LLM providers, agent frameworks and coding agents via auto-instrumentation — every call is captured as a span with no code changes beyond initialization. Don't see your integration? [Request one on GitHub](https://github.com/traceroot-ai/traceroot/issues/new). ## Native Auto-instrument with traceroot's official SDKs. ![](../logo/integrations/python.svg) ![](../logo/integrations/python-dark.svg) ### [Python SDK](/docs/tracing/python-sdk) ![](../logo/integrations/typescript.svg) ![](../logo/integrations/typescript-dark.svg) ### [TypeScript SDK](/docs/tracing/typescript-sdk) ## Frameworks Trace popular agent frameworks and orchestration libraries. ![](../logo/integrations/agno.png) ### [Agno](/docs/integrations/agno) ![](../logo/integrations/autogen.svg) ### [AutoGen](/docs/integrations/autogen) ![](../logo/integrations/claude-agent-sdk.svg) ### [Claude Agent SDK](/docs/integrations/claude-agent-sdk) ![](../logo/integrations/crewai.svg) ### [CrewAI](/docs/integrations/crewai) ![](../logo/integrations/dspy.png) ### [DSPy](/docs/integrations/dspy) ![](../logo/integrations/google-adk.png) ### [Google ADK](/docs/integrations/google-adk) ![](../logo/integrations/langchain.png) ### [LangChain & LangGraph](/docs/integrations/langchain) ![](../logo/integrations/langchain.png) ### [LangChain DeepAgents](/docs/integrations/langchain-deepagents) ![](../logo/integrations/llamaindex.png) ### [LlamaIndex](/docs/integrations/llamaindex) ![](../logo/integrations/mastra.svg) ![](../logo/integrations/mastra-dark.svg) ### [Mastra](/docs/integrations/mastra) ![](../logo/integrations/microsoft-agent-framework.svg) ### [Microsoft Agent Framework](/docs/integrations/microsoft-agent-framework) ![](../logo/integrations/openai-agents-sdk.svg) ![](../logo/integrations/openai-agents-sdk-dark.svg) ### [OpenAI Agents SDK](/docs/integrations/openai-agents-sdk) ![](../logo/integrations/pi.svg) ![](../logo/integrations/pi-dark.svg) ### [pi SDK](/docs/integrations/pi) ![](../logo/integrations/pydantic-ai.png) ### [Pydantic AI](/docs/integrations/pydantic-ai) ![](../logo/integrations/vercel-ai.svg) ![](../logo/integrations/vercel-ai-dark.svg) ### [Vercel AI SDK](/docs/integrations/vercel-ai) ## Model Providers Trace direct calls to LLM providers. ![](../logo/integrations/anthropic.svg) ![](../logo/integrations/anthropic-dark.svg) ### [Anthropic](/docs/integrations/anthropic) ![](../logo/integrations/gemini.svg) ### [Google Gemini](/docs/integrations/gemini) ![](../logo/integrations/mistral.svg) ### [Mistral](/docs/integrations/mistral) ![](../logo/integrations/openai.svg) ![](../logo/integrations/openai-dark.svg) ### [OpenAI](/docs/integrations/openai) ![](../logo/integrations/openai.svg) ![](../logo/integrations/openai-dark.svg) ### [OpenAI-Compatible](/docs/integrations/openai-compatible) ![](../logo/integrations/openrouter.svg) ![](../logo/integrations/openrouter-dark.svg) ### [OpenRouter](/docs/integrations/openrouter) ![](../logo/integrations/typesafe.svg) ### [TypeSafe AI](/docs/integrations/typesafe) ## Coding Agents Trace your coding agent's own sessions with traceroot's plugins and extensions. ![](../logo/integrations/claude-agent-sdk.svg) ### [Claude Code](/docs/integrations/claude-code) ![](../logo/integrations/openai.svg) ![](../logo/integrations/openai-dark.svg) ### [Codex](/docs/integrations/codex) ![](../logo/integrations/pi.svg) ![](../logo/integrations/pi-dark.svg) ### [pi CLI](/docs/integrations/pi-cli) ## Tools and APIs Report what paid tools and APIs cost on their own spans. ![](../logo/integrations/exa.png) ### [Exa](/docs/integrations/exa) ![](../logo/integrations/predictleads.png) ### [PredictLeads](/docs/integrations/predictleads) ![](../logo/integrations/tavily.svg) ### [Tavily](/docs/integrations/tavily) ![](../logo/integrations/tinyfish.png) ### [TinyFish](/docs/integrations/tinyfish) ## Internal Model Providers traceroot provides a set of built-in models that power the traceroot Agent — used for trace analysis, root cause analysis, and automated fix PR generation. No API key needed to get started. > [card] [Supported Models](/docs/integrations/internal-models) > > Browse the full list of built-in models, and learn how to use your own via > BYOK. # pi SDK Source: https://traceroot.ai/docs/integrations/pi Trace embedded pi coding-agent sessions with traceroot Automatically capture agent runs, LLM calls, tool executions, and token usage from [pi](https://pi.dev) coding-agent sessions you drive with `@earendil-works/pi-coding-agent` — the library form of pi, where you run an `AgentSession` from your own Node process instead of the interactive CLI. > [!NOTE] > > Running the interactive `pi` CLI instead? Use the [pi CLI](/docs/integrations/pi-cli) integration. ## Install > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot @earendil-works/pi-coding-agent > ``` ## Set up Call `TraceRoot.initialize()` before creating any session: ```typescript theme={null} import * as pi from '@earendil-works/pi-coding-agent'; import { TraceRoot } from '@traceroot-ai/traceroot'; TraceRoot.initialize({ instrumentModules: { piCodingAgent: pi }, }); ``` ## Usage Create and run sessions as usual — every session created after initialization is traced automatically: ```typescript theme={null} import { createAgentSession, ModelRegistry, ModelRuntime } from '@earendil-works/pi-coding-agent'; // pi 0.80's model + credential surface. Created async (it reads the on-disk // model catalog), then handed to createAgentSession. Injecting OPENAI_API_KEY // as a runtime credential authenticates from env alone — no `pi auth login`. const modelRuntime = await ModelRuntime.create(); await modelRuntime.setRuntimeApiKey('openai', process.env.OPENAI_API_KEY); const modelRegistry = new ModelRegistry(modelRuntime); const model = modelRegistry.find('openai', 'gpt-4o-mini'); const { session } = await createAgentSession({ model, modelRuntime }); try { await session.prompt('Write add.js exporting add(a, b), then write and run a test for it.'); } finally { session.dispose(); } ``` Each `session.prompt()` becomes a nested trace, with tool spans under the LLM span that requested them: ``` AgentSession.prompt # AGENT span — one per prompt() call └── gpt-4o-mini # LLM span — provider, tokens ├── write: add.js # TOOL span — arguments + result ├── write: test-add.js └── bash: node test-add.js ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `AgentSession.prompt`. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Agent runs | One span per `session.prompt()` call | | Model | Model id for each LLM call | | Messages | Prompt and assistant response, when content capture is enabled | | Tool calls | Each tool execution with arguments, result, and error status | | Tokens | Input, output, and cache read/write counts | | Latency | Duration per span | `captureContent` and `captureToolIo` (both default `true`) control whether prompt/response text and tool arguments/results are attached to spans. Pass them through the `{ module, config }` form: ```typescript theme={null} TraceRoot.initialize({ instrumentModules: { piCodingAgent: { module: pi, config: { captureContent: false, captureToolIo: false } }, }, }); ``` `captureContent` can also be a function that redacts, truncates, replaces, or omits each prompt and response before it is recorded. See [Data Capture and Privacy](/docs/tracing/data-capture) for a working redaction example. ## Limitations * `@earendil-works/pi-coding-agent` is ESM-only and requires Node >= 22.19. ## Run the example Clone the repo and run a complete session end-to-end. > [card] [TypeScript](https://github.com/traceroot-ai/traceroot/tree/main/examples/typescript/pi-agent-sdk) > > Run the TypeScript example ## Next steps > [card] [TypeScript SDK Reference](/docs/tracing/typescript-sdk) > > Explore the full TypeScript SDK reference and configuration options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # pi CLI Source: https://traceroot.ai/docs/integrations/pi-cli Trace interactive pi sessions with the traceroot extension Trace interactive [pi](https://pi.dev) sessions — turns, model calls, tool calls and token usage — with the [traceroot pi extension](https://github.com/traceroot-ai/traceroot-pi-extension). > [!NOTE] > > Embedding pi in your own Node process with `@earendil-works/pi-coding-agent`? Use the [pi SDK](/docs/integrations/pi) integration instead. ## Install Requires Node.js 22.19 or later and pi 0.79 or later. ```bash theme={null} pi install npm:@traceroot-ai/pi-extension ``` ## Set up Tracing is opt-in. Set these before starting pi, then restart pi: ```bash theme={null} export TRACEROOT_ENABLED=true export TRACEROOT_API_KEY="tr-..." # Optional: only needed for self-hosted deployments # export TRACEROOT_HOST_URL="https://your-traceroot.example.com" ``` | Setting | Env var | Default | Description | | - | - | - | - | | Enabled | `TRACEROOT_ENABLED` | `false` | Required. Turns tracing on. | | API key | `TRACEROOT_API_KEY` | — | Required. | | Host | `TRACEROOT_HOST_URL` | hosted service | Set only when self-hosting. | | Capture content | `TRACEROOT_CAPTURE_CONTENT` | `true` | Send prompts and responses. `false` keeps only metadata. | | Capture tool I/O | `TRACEROOT_CAPTURE_TOOL_IO` | `true` | Send tool arguments and results. | | Debug | `TRACEROOT_PI_DEBUG` | `false` | Log to stderr. `TRACEROOT_LOG_FILE` writes a JSON-lines log. | Inside pi, `/traceroot status` shows whether tracing is on and where it sends; `/traceroot disable` and `/traceroot enable` switch it for the current session. ## Usage Use pi as usual. Each session becomes one trace, with one span per turn. ``` pi.session # one per session ├── pi.turn # one per prompt │ ├── openai/gpt-… # LLM span — tokens, cache, cost │ │ ├── bash: … # TOOL span — command and result │ │ └── edit: … └── pi.turn ``` ## Verify Run one prompt in pi and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `pi.session`. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Sessions | One trace per session; resumed and forked sessions are linked | | Turns | One span per prompt, marked when interrupted | | Model calls | Provider and model, prompt and response; errors and aborts marked | | Tokens | Input, output, cache read and write counts, and cost | | Tool calls | Tool name, arguments and result; failed tools marked | | Compaction | Each compaction as a span on the session | With content capture on, prompts and tool I/O are sent as they are. Nothing is redacted; set `TRACEROOT_CAPTURE_CONTENT=false` and `TRACEROOT_CAPTURE_TOOL_IO=false` to send metadata only. See [Data Capture and Privacy](/docs/tracing/data-capture#what-is-captured-by-default). ## Limitations * The exporter ignores `HTTPS_PROXY`, so exports can fail on proxy-only networks. * If no trace arrives, run `/traceroot status` and set `TRACEROOT_PI_DEBUG=true` to log to stderr. ## Run the example > [card] [traceroot-pi-extension](https://github.com/traceroot-ai/traceroot-pi-extension) > > Source and full settings ## Next steps > [card] [pi SDK](/docs/integrations/pi) > > Trace pi sessions you run from your own code. > [card] [Data Capture and Privacy](/docs/tracing/data-capture) > > Control what leaves your machine. # PredictLeads Source: https://traceroot.ai/docs/integrations/predictleads Record the cost of PredictLeads lookups on their tool spans Record what each [PredictLeads](https://predictleads.com) company lookup costs next to your LLM spend. Every lookup becomes a tool span with its domain, the company data, latency and dollar cost, and the cost counts toward the trace total. > [!NOTE] > > traceroot does not auto-instrument PredictLeads and does not know your plan's price. You trace the lookup with `observe` and report its cost yourself, and traceroot stores the cost you report. See [Tool and API calls](/docs/tracing/cost-tracking#tool-and-api-calls) for how reported costs add up. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot httpx > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot > ``` Reporting a cost requires `traceroot` 0.5.0 or later, or `@traceroot-ai/traceroot` 0.7.0 or later. ## Set up Initialize traceroot once at startup. No integration flag is needed: the `observe` wrapper in the next step traces the lookup. > [tab] Python > > ```python theme={null} > import traceroot > > traceroot.initialize() > ``` > [tab] TypeScript > > ```typescript theme={null} > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize(); > ``` ## Usage PredictLeads charges a fixed price per request, set by your plan. To record it: 1. Wrap the lookup in a function traced as a tool span (`type="tool"`). 2. Check that the request succeeded; PredictLeads does not bill a rejected one. 3. Report your price per call with `update_current_span(cost=...)` (`updateCurrentSpan` in TypeScript) while that span is open. > [tab] Python > > ```python theme={null} > import os > > import httpx > from traceroot import observe, update_current_span > > PRICE_PER_CALL = 0.04 # your plan's price per request > HEADERS = { > "X-Api-Key": os.environ["PREDICTLEADS_API_KEY"], > "X-Api-Token": os.environ["PREDICTLEADS_API_TOKEN"], > } > > @observe(name="company_lookup", type="tool") > def company_lookup(domain: str): > response = httpx.get(f"https://predictleads.com/api/v3/companies/{domain}", headers=HEADERS) > if response.is_success: > update_current_span(cost=PRICE_PER_CALL) > return response.json() > ``` > [tab] TypeScript > > ```typescript theme={null} > import { observe, updateCurrentSpan } from '@traceroot-ai/traceroot'; > > const PRICE_PER_CALL = 0.04; // your plan's price per request > const headers = { > 'X-Api-Key': process.env.PREDICTLEADS_API_KEY!, > 'X-Api-Token': process.env.PREDICTLEADS_API_TOKEN!, > }; > > async function companyLookup(domain: string) { > return observe({ name: 'company_lookup', type: 'tool' }, async (d: string) => { > const response = await fetch(`https://predictleads.com/api/v3/companies/${d}`, { headers }); > if (response.ok) updateCurrentSpan({ cost: PRICE_PER_CALL }); > return response.json(); > }, domain); > } > ``` Call `company_lookup` from your agent like any other function. Its span nests under the span that called it, and its cost adds to that trace's total. ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a `company_lookup` span, at the root or under your agent's span. Open it to see its cost; the trace's total cost includes it. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Input | The domain passed to `company_lookup` | | Output | The company data returned | | Cost | Your price per call, in USD | | Latency | Request duration | ## Limitations * PredictLeads bills a lookup that returns no data yet, so report the cost on every successful response. * A retry is another call and gets its own cost. ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Fixed price per call](/docs/tracing/cost-tracking#fixed-price-per-call) > > Report your price on each billed call # Pydantic AI Source: https://traceroot.ai/docs/integrations/pydantic-ai Auto-instrument Pydantic AI agent runs, LLM calls, and tool invocations Automatically capture agent runs, LLM calls, and tool invocations within the [Pydantic AI](https://ai.pydantic.dev) framework via its native OpenTelemetry instrumentation. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot pydantic-ai > ``` ## Set up > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.PYDANTIC_AI]) > ``` ## Usage Once initialized, all Pydantic AI agent runs and tool invocations are captured automatically: ```python theme={null} import traceroot from traceroot import Integration from pydantic_ai import Agent traceroot.initialize(integrations=[Integration.PYDANTIC_AI]) agent = Agent( "openai:gpt-4o-mini", system_prompt="You are a helpful assistant.", ) result = agent.run_sync("What is the capital of France?") print(result.output) traceroot.flush() ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `invoke_agent agent`, named after the agent — Pydantic AI infers `agent` from the variable name unless you pass `Agent(name=...)`. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Agent runs | Each `agent.run()` / `agent.run_sync()` / `agent.run_stream()` invocation | | Tool calls | Each tool invocation with input arguments and results | | LLM calls | Raw completion requests with input messages, output messages, and model name | | Tokens & Cost | Input and output token counts; cost when the model ID matches traceroot's pricing table | | Cache tokens | Cache read and write token counts when reported by the provider | | Latency | Duration per agent run and per span | ## Limitations None known. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [Python](https://github.com/traceroot-ai/traceroot/tree/main/examples/python/pydantic-ai-tool-agent) > > Run the Python example ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # Tavily Source: https://traceroot.ai/docs/integrations/tavily Record the cost of Tavily searches on their tool spans Record what each [Tavily](https://tavily.com) search costs next to your LLM spend. Every search becomes a tool span with its query, results, latency and dollar cost, and the cost counts toward the trace total. > [!NOTE] > > traceroot does not auto-instrument Tavily and does not know your plan's price. You trace the search with `observe` and report its cost yourself, and traceroot stores the cost you report. See [Tool and API calls](/docs/tracing/cost-tracking#tool-and-api-calls) for how reported costs add up. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot tavily-python > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot @tavily/core > ``` Reporting a cost requires `traceroot` 0.5.0 or later, or `@traceroot-ai/traceroot` 0.7.0 or later. ## Set up Initialize traceroot once at startup. No integration flag is needed: the `observe` wrapper in the next step traces the search. > [tab] Python > > ```python theme={null} > import traceroot > > traceroot.initialize() > ``` > [tab] TypeScript > > ```typescript theme={null} > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize(); > ``` ## Usage Tavily bills in credits, and a credit's dollar price depends on your plan. It returns the credits a search used when you ask for usage. To record the cost: 1. Wrap the search in a function traced as a tool span (`type="tool"`). 2. Ask for usage with `include_usage=True` (`includeUsage: true` in TypeScript) and read the credits used. 3. Multiply by your price per credit and report it with `update_current_span(cost=...)` (`updateCurrentSpan` in TypeScript) while that span is open. > [tab] Python > > ```python theme={null} > from tavily import TavilyClient > from traceroot import observe, update_current_span > > PRICE_PER_CREDIT = 0.008 # your plan's price per credit > > client = TavilyClient() # reads TAVILY_API_KEY > > @observe(name="tavily_search", type="tool") > def web_search(query: str): > response = client.search(query, max_results=3, include_usage=True) > credits = response.get("usage", {}).get("credits") > if credits is not None: > update_current_span(cost=credits * PRICE_PER_CREDIT) > return response > ``` > [tab] TypeScript > > ```typescript theme={null} > import { tavily } from '@tavily/core'; > import { observe, updateCurrentSpan } from '@traceroot-ai/traceroot'; > > const PRICE_PER_CREDIT = 0.008; // your plan's price per credit > > const client = tavily({ apiKey: process.env.TAVILY_API_KEY }); > > async function webSearch(query: string) { > return observe({ name: 'tavily_search', type: 'tool' }, async (q: string) => { > const response = await client.search(q, { maxResults: 3, includeUsage: true }); > const credits = response.usage?.credits; > if (credits !== undefined) updateCurrentSpan({ cost: credits * PRICE_PER_CREDIT }); > return response; > }, query); > } > ``` Call `web_search` from your agent like any other function. Its span nests under the span that called it, and its cost adds to that trace's total. ![Span details for a tavily_search tool span with a cost of 0.008000 and its search query input](https://mintcdn.com/tracerootai/3SDwRl5xRxfZvr-4/images/tool_cost_credits_v1.png?fit=max\&auto=format\&n=3SDwRl5xRxfZvr-4\&q=85\&s=a399f51f86b80f63d676c4b3f47e2ccd) One credit at the plan rate on the tool span ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a `tavily_search` span, at the root or under your agent's span. Open it to see its cost; the trace's total cost includes it. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Input | The query passed to `web_search` | | Output | The search response | | Cost | Credits used × your price per credit, in USD | | Latency | Request duration | ## Limitations * Tavily returns credits only when the search asks for usage; without it the span records no cost. * Set `PRICE_PER_CREDIT` to your own plan's rate. ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Credits or quota](/docs/tracing/cost-tracking#credits-or-quota) > > Report credits used at your plan's price per credit # TinyFish Source: https://traceroot.ai/docs/integrations/tinyfish Record the cost of TinyFish browser sessions on their tool spans Record what each [TinyFish](https://tinyfish.ai) remote browser session costs next to your LLM spend. Every session becomes a tool span with its URL, result, duration and dollar cost, and the cost counts toward the trace total. > [!NOTE] > > traceroot does not auto-instrument TinyFish and does not know your rate. You trace the session with `observe` and report its cost yourself, and traceroot stores the cost you report. See [Tool and API calls](/docs/tracing/cost-tracking#tool-and-api-calls) for how reported costs add up. ## Install > [tab] Python > > ```bash theme={null} > pip install traceroot httpx > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot > ``` Reporting a cost requires `traceroot` 0.5.0 or later, or `@traceroot-ai/traceroot` 0.7.0 or later. ## Set up Initialize traceroot once at startup. No integration flag is needed: the `observe` wrapper in the next step traces the session. > [tab] Python > > ```python theme={null} > import traceroot > > traceroot.initialize() > ``` > [tab] TypeScript > > ```typescript theme={null} > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize(); > ``` ## Usage TinyFish bills remote browsers by the minute and returns no price, so you estimate the cost from the session's length. To record it: 1. Wrap the session in a function traced as a tool span (`type="tool"`). 2. Time the session from creation to close. 3. Multiply the minutes by your price per minute and report it with `update_current_span(cost=...)` (`updateCurrentSpan` in TypeScript) before the span closes. A `finally` block reports it even when the session fails. > [tab] Python > > ```python theme={null} > import os > import time > > import httpx > from traceroot import observe, update_current_span > > PRICE_PER_MINUTE = 0.002 # your price per browser minute > API = "https://api.browser.tinyfish.ai" > HEADERS = {"X-API-Key": os.environ["TINYFISH_API_KEY"]} > > @observe(name="tinyfish_open_page", type="tool") > def browse(url: str): > start = time.monotonic() > response = httpx.post(API, headers=HEADERS, json={"url": url}, timeout=60) > response.raise_for_status() > session = response.json() > try: > ... # drive the browser through session["cdp_url"] > finally: > minutes = (time.monotonic() - start) / 60 > update_current_span(cost=minutes * PRICE_PER_MINUTE) > httpx.delete(f"{API}/{session['session_id']}", headers=HEADERS) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { observe, updateCurrentSpan } from '@traceroot-ai/traceroot'; > > const PRICE_PER_MINUTE = 0.002; // your price per browser minute > const API = 'https://api.browser.tinyfish.ai'; > const headers = { 'X-API-Key': process.env.TINYFISH_API_KEY!, 'Content-Type': 'application/json' }; > > async function browse(url: string) { > return observe({ name: 'tinyfish_open_page', type: 'tool' }, async (u: string) => { > const start = performance.now(); > const created = await fetch(API, { method: 'POST', headers, body: JSON.stringify({ url: u }) }); > if (!created.ok) throw new Error(`TinyFish session failed: ${created.status}`); > const session = await created.json(); > try { > // drive the browser through session.cdp_url > } finally { > const minutes = (performance.now() - start) / 60000; > updateCurrentSpan({ cost: minutes * PRICE_PER_MINUTE }); > await fetch(`${API}/${session.session_id}`, { method: 'DELETE', headers }); > } > }, url); > } > ``` Call `browse` from your agent like any other function. Its span nests under the span that called it, and its cost adds to that trace's total. ![Span details for a tinyfish_open_page tool span with a cost of 0.000092, the page URL input and the loaded page title output](https://mintcdn.com/tracerootai/3SDwRl5xRxfZvr-4/images/tool_cost_time_used_v1.png?fit=max\&auto=format\&n=3SDwRl5xRxfZvr-4\&q=85\&s=1926ae586939d607f3710c4eabc5fa46) Browser minutes at the per-minute rate on the tool span ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a `tinyfish_open_page` span, at the root or under your agent's span. Open it to see its cost; the trace's total cost includes it. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Input | The URL passed to `browse` | | Output | What `browse` returns, such as the page title | | Cost | Session minutes × your price per minute, in USD | | Latency | Session duration | ## Limitations * The cost is an estimate: TinyFish bills from its own usage records, which can run a little longer. ## Next steps > [card] [Python SDK Reference](/docs/tracing/python-sdk) > > Explore the full Python SDK reference and instrumentation options. > [card] [Time used](/docs/tracing/cost-tracking#time-used) > > Report time used at your price per minute # TypeSafe AI Source: https://traceroot.ai/docs/integrations/typesafe Auto-instrument Jev calls made with the TypeSafe AI SDK Automatically capture every call to Jev, TypeSafe AI's System One decision model. Each `systemOne` call becomes a decision span with the model version, token usage, cost, and the questions, answers and probabilities. ## Setup > [tab] TypeScript > > ```typescript theme={null} > import * as typeSafe from '@typesafe-ai/sdk'; > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize({ instrumentModules: { typeSafe } }); > ``` Requires `@traceroot-ai/traceroot` 0.5.0 or newer and Node.js 20 or newer. ## Usage Once initialized, all `systemOne` calls are captured automatically: > [tab] TypeScript > > ```typescript theme={null} > import { choice, TypeSafeClient } from '@typesafe-ai/sdk'; > > const client = new TypeSafeClient(); // reads TYPESAFE_API_KEY > > // This call is automatically traced > const response = await client.systemOne({ > state: { document: 'I was charged twice. Please fix this ASAP.' }, > questions: { > category: choice('What is this ticket about?', { > billing: null, > technical: null, > other: null, > }), > }, > }); > > console.log(response.answers.category.choice); > ``` ## What Gets Captured | Attribute | Description | | - | - | | Model | The resolved Jev version, e.g. `jev-1.13.0` | | Input | The `state` and `questions` sent to Jev | | Output | Answers and their probabilities | | Tokens | Input and output tokens | | Cost | Calculated when the model ID matches traceroot's pricing table | | Latency | Request duration | Jev calls are recorded as decision spans, not LLM spans, because Jev selects among the options you give it instead of generating text. # Vercel AI SDK Source: https://traceroot.ai/docs/integrations/vercel-ai Trace Vercel AI SDK agents — no instrumentModules config required Automatically capture agent runs, LLM calls, tool executions, and token usage from the [Vercel AI SDK](https://ai-sdk.dev) through its built-in OpenTelemetry support — no `instrumentModules` config required. ## Install > [tab] AI SDK 7 > > ```bash theme={null} > npm install @traceroot-ai/traceroot ai@^7 @ai-sdk/otel@^1 @ai-sdk/openai@^4 zod@^4 > ``` > [tab] AI SDK 6 (legacy) > > ```bash theme={null} > npm install @traceroot-ai/traceroot ai@^6 @ai-sdk/openai@^3 zod@^4 > ``` ## Set up > [tab] AI SDK 7 > > Call `TraceRoot.initialize()` and register the AI SDK's OpenTelemetry integration before your first model call: > > ```typescript theme={null} > import { TraceRoot } from '@traceroot-ai/traceroot'; > import { registerTelemetry } from 'ai'; > import { OpenTelemetry } from '@ai-sdk/otel'; > > TraceRoot.initialize(); > > // Required on AI SDK 7 — without it, no spans are emitted. > registerTelemetry(new OpenTelemetry({ usage: true })); > ``` > [tab] AI SDK 6 (legacy) > > Call `TraceRoot.initialize()` before your first model call: > > ```typescript theme={null} > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize(); > ``` > > Then set `experimental_telemetry: { isEnabled: true }` on each call (see [Usage](#usage)). ## Usage Once initialized, every `generateText` / `streamText` call, model call, and tool execution is captured automatically: > [tab] AI SDK 7 > > ```typescript theme={null} > import { TraceRoot } from '@traceroot-ai/traceroot'; > import { registerTelemetry, generateText, tool, isStepCount } from 'ai'; > import { OpenTelemetry } from '@ai-sdk/otel'; > import { openai } from '@ai-sdk/openai'; > import { z } from 'zod'; > > TraceRoot.initialize(); > registerTelemetry(new OpenTelemetry({ usage: true })); > > // The agent run, both model calls, and the getWeather tool call are all traced. > const result = await generateText({ > model: openai('gpt-4o-mini'), > instructions: 'You are a concise weather assistant.', > prompt: 'What is the weather in Tokyo?', > stopWhen: isStepCount(5), > telemetry: { functionId: 'weather_agent' }, > tools: { > getWeather: tool({ > description: 'Get current weather for a city', > inputSchema: z.object({ city: z.string() }), > execute: async ({ city }) => ({ city, temp: 72, condition: 'sunny' }), > }), > }, > }); > > console.log(result.text); > await TraceRoot.flush(); > ``` > [tab] AI SDK 6 (legacy) > > ```typescript theme={null} > import { TraceRoot } from '@traceroot-ai/traceroot'; > import { generateText, tool, stepCountIs } from 'ai'; > import { openai } from '@ai-sdk/openai'; > import { z } from 'zod'; > > TraceRoot.initialize(); > > // The agent run, both model calls, and the getWeather tool call are all traced. > const result = await generateText({ > model: openai('gpt-4o-mini'), > system: 'You are a concise weather assistant.', > prompt: 'What is the weather in Tokyo?', > stopWhen: stepCountIs(5), > // Required on every call in AI SDK 6. > experimental_telemetry: { isEnabled: true, functionId: 'weather_agent' }, > tools: { > getWeather: tool({ > description: 'Get current weather for a city', > inputSchema: z.object({ city: z.string() }), > execute: async ({ city }) => ({ city, temp: 72, condition: 'sunny' }), > }), > }, > }); > > console.log(result.text); > await TraceRoot.flush(); > ``` ## Verify Run your code and confirm the trace arrives: 1. **In the traceroot UI**: Navigate to **Tracing > Traces** in [traceroot Cloud](https://app.traceroot.ai). You should see a trace with root span `ai.generateText`. `functionId` sets a telemetry attribute on the span; it does not rename it. 2. **Via the CLI**: Run `traceroot traces list` to inspect your recent traces from the terminal. ## What gets captured | Attribute | Description | | - | - | | Model | The model passed to `generateText` / `streamText` / `ToolLoopAgent` | | Messages | Input prompt and message history | | Response | Generated text output | | Tool calls | Each tool invocation with input and output | | Tokens | Input and output token counts | | Cost | Calculated when the model ID matches traceroot's pricing table | | Latency | Request duration per span | | Multi-step iterations | Each step of a tool loop captured as a child span | ## Limitations * AI SDK 7 requires Node >= 22. * On AI SDK 6 (legacy), each model call must explicitly set `experimental_telemetry: { isEnabled: true }`. ## Embedding usage `embed` and `embedMany` record reported input-token usage on their provider-call spans. Cost is calculated when the model matches an entry in traceroot's pricing table; otherwise, cost remains unset. The current standard pricing table has no embedding models. The operation wrapper remains in the trace, but its aggregate legacy usage is not counted again. This also applies when a batch is split across multiple provider calls. Embedding spans without reported usage keep their token counts unset. traceroot does not estimate embedding tokens from input text or output vectors, avoiding duplicate estimates on orchestration spans whose provider-call children report the actual usage. For AI SDK 7's `OpenTelemetry` integration, use `@ai-sdk/otel` version 1.0.76 or newer. If you use an older version, upgrade `@ai-sdk/otel`: older semconv emitters can repeat usage on both the embedding wrapper and its provider calls, while the fixed emitter reports usage only on provider calls. AI SDK 6's built-in telemetry and AI SDK 7's `LegacyOpenTelemetry` integration report raw embedding usage as `ai.usage.tokens`. traceroot reads this field only on provider calls identified by `ai.operationId` or `operation.name` as `ai.embed.doEmbed` or `ai.embedMany.doEmbed`. Preserve these operation attributes in custom span processors so usage can be distinguished from wrapper aggregates. If your emitter omits them, use an emitter that supplies them or switch to the current AI SDK 7 `OpenTelemetry` integration. Normalized OpenInference or GenAI usage fields continue to be supported independently. ## Run the example Clone the repo and run a complete agent end-to-end. > [card] [TypeScript](https://github.com/traceroot-ai/traceroot/tree/main/examples/typescript/vercel-ai) > > Run the TypeScript example ## Next steps > [card] [TypeScript SDK Reference](/docs/tracing/typescript-sdk) > > Explore the full TypeScript SDK reference and configuration options. > [card] [Cost Tracking](/docs/tracing/cost-tracking) > > Learn how traceroot calculates token costs and tracks model spend. # Track recurring agent failures with signals Source: https://traceroot.ai/docs/signals/introduction See which traces share a problem and how often it still occurs after a fix. A signal is one problem your agent keeps running into. When a detector flags many traces for the same problem, traceroot groups them into one signal. You review and fix the problem once, then see how often it still happens. Signals is an extension of [detectors](/docs/detectors/introduction), which judge traces; a signal follows one problem across many traces. Signals work on every plan, including Free and self-hosted. Root cause analysis is included to run 30 analyses on Free, 100 per billing period on Starter and Pro (extra analyses are billed), and unlimited on Enterprise and self-hosted. ![Signals list with one signal's panel open, showing its status, summary, root cause analysis and count over time](https://mintcdn.com/tracerootai/9GPqEa37t7x_ANSu/images/signals_page_v1.png?fit=max\&auto=format\&n=9GPqEa37t7x_ANSu\&q=85\&s=38b802b388fff3cc52b91be5dcacd4f1) Signals page: the signal list, then one signal's status, summary and analysis ## How findings become signals When a detector flags a trace, traceroot records a **finding**. Within about a minute, traceroot adds the finding to the detector's matching signal, or starts a new signal with a name and a **Summary**. A detector on a [TypeSafe AI](/docs/ai-agent/byok) model groups its findings by the category the model picked. To stop grouping for a detector, turn off **Generate signals** in the **Signals** section of its settings. The detector still records findings, but traceroot does not group, analyze or announce them. ## Review signals Open your project in [traceroot Cloud](https://app.traceroot.ai) and select **Signals** from the left nav. The list shows the newest signals first, filtered to **Open**. You can filter by status, by signal name or ID, or by detector name or ID. On the **Detectors** page, a detector's **Signals** count opens this list filtered to that detector. That count, and the list it opens, cover signals in every status. The date range (**Last 7 days** by default) sets the period for each signal's count of affected traces. It changes only the trace counts, and the chart and trace list in a signal's panel. It never hides a signal. Select a signal to open its panel: * **Status**: the signal's state, and the control to change it. * **Summary**: what the signal covers. * **Root Cause Analysis**: the latest analysis, or a **Run root cause analysis** button. When an analysis is shown, **Open in agent** opens it in the traceroot Agent for follow-up questions. * **Count over time**: of the traces the detector checked, how many this signal affected (**Affected**) and how many it did not (**Unaffected**), over the date range. * **Affected traces**: up to 10 recent traces from the date range. **View all** opens them on the Tracing page, filtered to the signal. In notification emails and Slack messages, a signal's name links straight to its panel. ## Signal states A signal is **Open**, **Resolved** or **Dismissed**. Change it with the **Status** control on its panel. * **Open**: the problem needs attention. Every new signal starts here. * **Resolved**: the problem is fixed. Pick a reason, such as **PR was merged**. * **Dismissed**: the signal needs no action. Pick a reason, such as **Expected behavior**. A signal keeps collecting new findings in every state. [Notifications](/docs/detectors/introduction#notifications) announce only new and reopened signals. A state changes as follows: * A resolved signal reopens by itself when a finding arrives from a trace that started after you resolved it. The next notification lists it as **Reopened**. * A dismissed signal stays dismissed. * Setting a signal back to **Open** by hand sends no notification and starts no analysis. ## Signals in the API Scripts can read signals and change their status through the public API: * `GET /api/v1/public/signals` lists the project's signals, newest first. Filter by `status` or `detector_id`. Merged signals are left out. * `GET /api/v1/public/signals/{signal_id}` returns one signal. A merged signal names the signal it moved to in `merged_into_id`. * `PATCH /api/v1/public/signals/{signal_id}/status` resolves or reopens a signal. Resolving needs a `reason` (`fixed_elsewhere`, `fixed_by_pr`, `already_fixed_before` or `other`), and `other` also needs a `note`. Send `expected_status` to refuse the change when someone changed the status after you read it. The [traceroot CLI](/docs/cli/get-started) does not have these signal commands yet. For user-token authentication, include `project_id` as a query parameter on GET requests and in the JSON body on PATCH requests. Listing returns 50 signals by default; increment `page` (starting at 0) to read subsequent pages. A project API key can read signals but cannot change them. A status change needs a signed-in user with the member role or higher, and it is recorded with that user's name, reason and note, as in the web app. You cannot dismiss a signal through the API: a dismissed signal never reopens, so a person dismisses it in the web app. ## Root cause analysis The [traceroot Agent](/docs/ai-agent/root-cause-analysis) reads one of the signal's traces and explains the cause. With [GitHub](/docs/ai-agent/github-integration) connected, it also reads the related code and recent commits. The detector's **Root cause analysis** setting decides when it runs: * **Automatic**: on each new signal, and on a reopened signal when the last analysis is more than 24 hours old. * **Manual**: only when you click **Run root cause analysis** on the signal's panel. A run starts within about two minutes. ## Next steps > [card] [Detectors](/docs/detectors/introduction) > > Set up the checks that flag problem traces. > [card] [Detector Notifications](/docs/detectors/introduction#notifications) > > Email and Slack digests for new and reopened signals. # Cost Tracking Source: https://traceroot.ai/docs/tracing/cost-tracking Automatic token counting and cost calculation for LLM calls traceroot captures token usage from instrumented LLM calls. It calculates cost when the reported model ID matches an entry in traceroot's pricing table, regardless of which integration emitted the span, or uses the cost you [report directly](#report-cost-directly). > [tab] Python > > ```python theme={null} > from openai import OpenAI > > import traceroot > from traceroot import Integration > > traceroot.initialize(integrations=[Integration.OPENAI]) > client = OpenAI() > > # OpenAI calls are tracked automatically; cost requires a priced model ID > client.chat.completions.create( > model="gpt-4o", > messages=[{"role": "user", "content": "Hello"}], > ) > ``` > [tab] TypeScript > > ```typescript theme={null} > import OpenAI from 'openai'; > import { TraceRoot } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize({ instrumentModules: { openAI: OpenAI } }); > const client = new OpenAI(); > > // OpenAI calls are tracked automatically; cost requires a priced model ID > await client.chat.completions.create({ > model: 'gpt-4o', > messages: [{ role: 'user', content: 'Hello' }], > }); > ``` ## How It Works 1. **Auto-instrumentation** — traceroot integrations capture token usage from the provider response automatically. 2. **Fallback estimation** — When token counts aren't in the response, traceroot estimates using tiktoken. 3. **Model matching** — The reported model ID is matched against traceroot's pricing table, including supported provider-prefixed and hosted aliases. 4. **Cost calculation** — When a match exists, token counts are multiplied by that entry's prices. An unmatched model still records tokens but has no calculated cost unless you [report one](#report-cost-directly). 5. **Reported cost** — If the span carries a cost you reported, it is stored in place of the calculated cost from step 4. Tokens are unchanged. ## Pricing coverage Pricing entries currently cover these model families: * OpenAI * Anthropic, including supported model IDs routed through Amazon Bedrock and Claude 3.x model IDs on Google Vertex AI * Google Gemini * xAI * DeepSeek * Moonshot (Kimi) * Z.AI (GLM) * TypeSafe AI (JEV) Coverage is based on model-ID patterns, not merely the provider or integration name. Notable model families without pricing entries include Mistral, Groq-hosted models, Amazon Nova and Titan, and Meta Llama. Routed models, such as models called through OpenRouter or Bedrock, are priced only when the routed model ID matches a pricing entry. If you call these through your own instrumented code, you can [report the cost directly](#report-cost-directly). ## What Gets Tracked Each LLM span captures: | Metric | Description | | - | - | | Input tokens | Tokens in the prompt/messages | | Output tokens | Tokens in the completion/response | | Total tokens | Sum of input and output | | Cost (USD) | Calculated when the model ID matches the pricing table, or the cost you report | ## Viewing Costs In the dashboard, costs are visible at both the span and trace level: * **Per span** — see the cost of each individual LLM, tool or API call * **Per trace** — see the total cost aggregated across all calls in the trace ![Trace detail with the LLM span cost chip and its Cost breakdown card highlighted](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/cost_breakdown_v1.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=30f66a56292555aaaaba61749f0cf2d7) Span cost: cost chip and its input and output cost breakdown ## Manual Token Reporting For custom or unsupported providers, report token usage manually: > [tab] Python > > ```python theme={null} > from traceroot import observe, update_current_span > > class MockLLM: > def generate(self, prompt): > class Response: > text = "mock response" > input_token_count = 10 > output_token_count = 20 > return Response() > > my_llm = MockLLM() > > @observe(name="custom_llm", type="llm") > def call_custom_llm(prompt: str): > response = my_llm.generate(prompt) > > update_current_span( > model="claude-opus-5", > usage={ > "input_tokens": response.input_token_count, > "output_tokens": response.output_token_count, > }, > ) > > return response.text > ``` > [tab] TypeScript > > ```typescript theme={null} > import { observe, updateCurrentSpan } from '@traceroot-ai/traceroot'; > > const myLlm = { > generate: async (prompt: string) => ({ > text: 'mock response', > inputTokenCount: 10, > outputTokenCount: 20, > }), > }; > > const result = await observe({ name: 'custom_llm', type: 'llm' }, async () => { > const prompt = "hello"; > const response = await myLlm.generate(prompt); > > updateCurrentSpan({ > model: 'claude-opus-5', > usage: { > input: response.inputTokenCount, > output: response.outputTokenCount, > }, > }); > > return response.text; > }); > ``` > [!NOTE] > > **Usage keys differ by language:** > In Python, use `input_tokens` and `output_tokens` (and `reasoning_tokens`, `cache_read_tokens`, `cache_write_tokens`). > In TypeScript, use `input`, `output`, `cacheRead`, `cacheWrite`, and `total`. ## Report cost directly When you already know what a call cost (a negotiated rate, a gateway's bill, an unpriced model), report it with the usage: > [tab] Python > > ```python theme={null} > from traceroot import update_current_span > > update_current_span( > model="claude-opus-5", > usage={"input_tokens": 10, "output_tokens": 20}, > cost=0.0123, # USD > ) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { updateCurrentSpan } from '@traceroot-ai/traceroot'; > > updateCurrentSpan({ > model: 'claude-opus-5', > usage: { input: 10, output: 20 }, > cost: 0.0123, // USD > }); > ``` > [!NOTE] > > Requires the Python SDK `traceroot` 0.5.0 or later, or the TypeScript SDK `@traceroot-ai/traceroot` 0.7.0 or later. A reported cost replaces traceroot's computed price for that span. Report it as a number on the span that made the call. A cost of `0` is accepted and sets the span's cost to zero. Reporting a total on a parent span whose child spans are also priced counts the cost twice. Tokens are still recorded. The per-category cost breakdown is shown only when it matches the computed price, so a different reported cost hides it for that span and its trace. The cost is read from the OpenInference `llm.cost.total` attribute, so an OpenInference instrumentor that sets it is picked up automatically. ### Tool and API calls A paid tool or API call takes a cost the same way, with no model or token usage needed. Tools bill in different ways, so first work out the dollar amount for the call, then report it on the tool's own span. The cost counts toward the trace total. | How the tool bills | What to report | Example | | - | - | - | | [Price in the response](#price-in-the-response) | The price from the response | [Exa](/docs/integrations/exa) | | [Fixed price per call](#fixed-price-per-call) | Your price per call | [PredictLeads](/docs/integrations/predictleads) | | [Credits or quota](#credits-or-quota) | Credits used × your price per credit | [Tavily](/docs/integrations/tavily) | | [Time used](#time-used) | Minutes used × your price per minute | [TinyFish](/docs/integrations/tinyfish) | #### Price in the response Some providers, such as [Exa](/docs/integrations/exa), return the dollar cost of each request. Pass it through; the Exa page shows the code. Some providers settle the price after the call returns. [Apify](https://docs.apify.com/api/v2/actor-run-get), for example, recommends reading a run's cost again about 10 seconds after it finishes. Report the settled value. #### Fixed price per call When every call costs the same, report your price on each call. A retry is another call and gets its own cost. traceroot records the cost you report, so report it for every response your provider bills. [PredictLeads](/docs/integrations/predictleads), for example, bills a lookup that returns no data yet but not a rejected one. #### Credits or quota Some providers bill in credits, and a credit's dollar price depends on your plan. [Tavily](/docs/integrations/tavily) returns the credits a search used when you ask for usage. #### Time used Some tools bill by time, such as a remote browser billed per minute. Time the session and multiply by your price per minute, and report it in a `finally` block so a failed session is still billed. This is an estimate: the provider bills from its own usage records, which can run a little longer. [TinyFish](/docs/integrations/tinyfish) browser sessions work this way. # Data Capture and Privacy Source: https://traceroot.ai/docs/tracing/data-capture Understand and control what traceroot records and exports traceroot spans can contain application data. Review the defaults before tracing workloads that process personal data, credentials, or confidential content. ## What is captured by default | Data | Default | How to limit it | | - | - | - | | Observed function arguments | Captured as span input | Set `capture_input=False` in Python or `captureInput: false` in TypeScript | | Observed function return values | Captured as span output | Set `capture_output=False` in Python or `captureOutput: false` in TypeScript | | Auto-instrumented LLM prompts and completions | Captured when the integration exposes content | Rewrite or blank values with [`redact`](#redact-personal-data-before-export); the TypeScript Pi integrations also expose a content filter (`captureContent`) | | Tool arguments and results | Captured when the integration exposes tool I/O | Rewrite or blank values with [`redact`](#redact-personal-data-before-export); the TypeScript Pi integrations also expose a tool I/O capture option (`captureToolIo: false`) | | Git repository and ref | Trace-level metadata when resolved | Automatically resolved even when not explicitly provided; there is no git-context opt-out. Disable tracing entirely to prevent export; see [Git Context](/docs/tracing/git-context) | | Source file, line, and function | Added by `observe` / `@observe` | Disable tracing entirely if source location must not leave the process | Capture settings prevent the selected values from being written to spans. They do not modify what your application passes to an LLM provider or tool. ## Disable input and output for one function > [tab] Python > > ```python theme={null} > from traceroot import observe > > @observe(capture_input=False, capture_output=False) > def verify_password(username: str, password: str) -> bool: > return check_credentials(username, password) > ``` > [tab] TypeScript > > Pass the function's arguments after it so `observe` can invoke it; `captureInput: false` keeps those arguments off the span: > > ```typescript theme={null} > import { observe } from '@traceroot-ai/traceroot'; > > const authenticated = await observe( > { name: 'verify_password', captureInput: false, captureOutput: false }, > verifyPassword, > username, > password, > ); > ``` These flags apply only to the span created by that `observe` call. Auto-instrumented child calls can still capture their own inputs and outputs according to the integration's settings. ## Redact personal data before export Requires `traceroot` 0.4.0 or later (Python) or `@traceroot-ai/traceroot` 0.6.0 or later (TypeScript, or 0.8.0 for an async `redact`). Pass a `redact` function to `initialize()`. It receives each finished span's attributes and returns what to export, so anything it removes never reaches traceroot. This example blanks email addresses and phone numbers: > [tab] Python > > ```python theme={null} > import re > import traceroot > > PII = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+|(? > def clean(value): > if isinstance(value, str): > return PII.sub("[REDACTED]", value) > if isinstance(value, (list, tuple)): > return [PII.sub("[REDACTED]", v) if isinstance(v, str) else v for v in value] > return value > > def redact(attributes: dict) -> dict: > return {key: clean(value) for key, value in attributes.items()} > > traceroot.initialize(redact=redact) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { TraceRoot } from '@traceroot-ai/traceroot'; > > const PII = /[\w.+-]+@[\w-]+\.[\w.]+|(? > const clean = (value: string) => value.replace(PII, '[REDACTED]'); > > TraceRoot.initialize({ > redact: (attributes) => > Object.fromEntries( > Object.entries(attributes).map(([key, value]) => [ > key, > typeof value === 'string' > ? clean(value) > : Array.isArray(value) && value.every((v): v is string => typeof v === 'string') > ? value.map(clean) > : value, > ]), > ), > }); > ``` How it works: * It runs in your app on every finished span, including spans from other libraries. In Python, keep it fast, synchronous, and thread-safe. In TypeScript, it runs at export, not during the request, and may be async; each batch gets 5 seconds. * It receives span content: `observe` input, output, and metadata, LLM prompts, event attributes such as exception messages, and the status message. Structured values arrive as JSON text, so parse them if your detector misses names in JSON. * It does not receive bookkeeping: user and session IDs, model name and token usage, tags, environment, span names, and similar. Those are sent as given, so keep personal data out of them and use an internal user ID, not an email address. * Return every key you want to keep. * Only traceroot's copy changes. Other exporters in your app still get the original span. > [!WARNING] > > If the function raises or returns anything other than a dict (Python) or object (TypeScript), metadata is dropped and every value except numbers and booleans is sent as `[redaction failed]`. A TypeScript promise that rejects or does not settle within 5 seconds fails the same way. The SDK logs a warning. In TypeScript, pass `logLevel: 'warn'` to `initialize()` to see it. Eval results are redacted too. Datasets and scorer definitions are not, so keep personal data out of them. A regex will not catch names or addresses. For those, call a detection service you run, for example a small one in front of [Presidio](https://microsoft.github.io/presidio/). This example sends each string to a service that takes and returns `{ text }`: ```typescript theme={null} import { TraceRoot } from '@traceroot-ai/traceroot'; TraceRoot.initialize({ redact: async (attributes) => { const detect = async (text: string): Promise => { const response = await fetch(process.env.PII_SERVICE_URL!, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ text }), signal: AbortSignal.timeout(4000), }); if (!response.ok) throw new Error(`PII service returned ${response.status}`); return ((await response.json()) as { text: string }).text; }; const entries = Object.entries(attributes).map(async ([key, value]) => [ key, typeof value === 'string' ? await detect(value) : Array.isArray(value) && value.every((v): v is string => typeof v === 'string') ? await Promise.all(value.map(detect)) : value, ]); return Object.fromEntries(await Promise.all(entries)); }, }); ``` > [!NOTE] > > From 0.8.0, TypeScript `redact` also receives keys derived from Vercel AI SDK attributes (`input.value`, `output.value`, `llm.*`), so if you drop specific `ai.*` keys, drop these too. The trace view marks every span the function ran on as **Redacted by your app**. ![Span details for update_shipping_contact with a Redacted by your app label, and an input message whose email address and phone number show as \[REDACTED\]](https://mintcdn.com/tracerootai/kqejnfY8AIiTJVnt/images/span_redacted_v1.png?fit=max\&auto=format\&n=kqejnfY8AIiTJVnt\&q=85\&s=5e5d33471e6a4855d9a2d4bf1cef845f) Span details: the Redacted by your app label and the redacted input ## Redact Pi content before export The TypeScript Pi instrumentation accepts a `captureContent` callback. It runs before content is attached to a span and can return replacement text or `undefined` to omit that item. ```typescript theme={null} import * as pi from '@earendil-works/pi-coding-agent'; import { TraceRoot, type ContentCaptureFn } from '@traceroot-ai/traceroot'; const EMAIL = /[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}/gi; const redactContent: ContentCaptureFn = (kind, value) => { if (kind === 'agent_input') { return value.text?.replace(EMAIL, '[redacted-email]'); } // Do not record model inputs or response bodies. return undefined; }; TraceRoot.initialize({ instrumentModules: { piCodingAgent: { module: pi, config: { captureContent: redactContent, captureToolIo: false, }, }, }, }); ``` The callback form is available in the TypeScript Pi instrumentation. For other integrations and for Python, use [`redact`](#redact-personal-data-before-export) above. ## Disable tracing entirely Set `TRACEROOT_ENABLED` before the SDK initializes: ```bash theme={null} TRACEROOT_ENABLED=false ``` The accepted values differ by SDK: | SDK | Values treated as off | | - | - | | Python | `false`, `0`, `no`, or `off`, case-insensitive | | TypeScript | The exact lowercase string `false` | You can also pass `enabled=False` to Python's `initialize()` or `enabled: false` to `TraceRoot.initialize()` in TypeScript. ## Privacy checklist * Classify prompts, completions, function arguments, tool I/O, metadata, tags, and source paths before enabling tracing in production. * Disable capture as close as possible to the function or integration that handles sensitive data. * Use `redact` to strip personal data that has to stay in captured content. * Never attach secrets to metadata or manual span updates. * Verify a representative exported trace in a non-production project before rollout. * Apply your organization's retention, access-control, and data-processing requirements to the traceroot deployment you use. # Set up tracing for your AI agent Source: https://traceroot.ai/docs/tracing/get-started Run the CLI from your repository, or instrument your app with the Python or TypeScript SDK. Run this from the root of the repository you want to trace: ```bash theme={null} npx -y traceroot-cli@latest setup ``` Setup needs Node.js 20.12 or later and a Python or TypeScript/JavaScript application. Using a coding agent? [Coding agents](/docs/getting-started/coding-agents#set-up-tracing) has a prompt to paste. Prefer to add the SDK yourself? Skip to [Set up manually](#set-up-manually). ## What setup does Setup checks for uncommitted changes, signs you in through your browser, selects or creates a project, and sets `TRACEROOT_API_KEY` for your application, normally in `.env.local` or `.env`. It then runs your coding agent and waits for a trace, or gives you a task or manual steps to finish yourself. Setup never reverts your coding agent's edits, so review them with `git diff`. It keeps its progress and a project key for the CLI in `.traceroot/`, which it git-ignores. In a monorepo, run setup from the service's directory or pass `--service `. For every option, run `npx -y traceroot-cli@latest setup --help`. Your deployed application also needs `TRACEROOT_API_KEY` in its environment, plus `TRACEROOT_HOST_URL` if you self-host. ## Self-hosted traceroot Pass your API's origin as `--host`, and your web app's origin, where you sign in, as `--auth-host`. For a local deployment with the default ports: ```bash theme={null} npx -y traceroot-cli@latest setup --host "http://localhost:8000" --auth-host "http://localhost:3000" ``` Use origins only, without `/api/v1`. If both share one origin, `--host` alone is enough. Your deployment's **Traces** page shows this command with its origins filled in. See [Self-Hosting](/docs/developer/self-hosting) to run a deployment. ## Set up manually These are the steps setup automates. ### 1. Get an API key Create an API key either with [traceroot Cloud](https://app.traceroot.ai) or [Self-Hosting](/docs/developer/self-hosting). ### 2. Install the SDK > [tab] Python > > ```bash theme={null} > pip install traceroot openai > ``` > [tab] TypeScript > > ```bash theme={null} > npm install @traceroot-ai/traceroot openai > ``` ### 3. Set environment variables ![traceroot Tracing page with the Create an API key step highlighted](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/tracing_api_key_v1.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=230ed774e147fbc2ea58bd028f582e35) Tracing setup: sidebar, Manual tab, API key field, Generate button Neither SDK loads `.env` files automatically; both read the process environment. Export the variables in your shell: ```bash theme={null} export TRACEROOT_API_KEY="your-api-key" export TRACEROOT_HOST_URL="https://app.traceroot.ai" # self-hosted: your API's origin, e.g. http://localhost:8000 ``` *(If you prefer to use a `.env` file, ensure you load it explicitly in your code as shown in the next step.)* ### 4. Instrument your code > [tab] Python > > Initialize traceroot where your application starts, inside the `if __name__ == "__main__":` block rather than at module scope, so a test that imports the module doesn't send traces. > > ```python theme={null} > # from dotenv import load_dotenv; load_dotenv() # Uncomment if using a .env file (requires: pip install python-dotenv) > from openai import OpenAI > > import traceroot > from traceroot import Integration > > # This single line instruments all OpenAI API calls > traceroot.initialize(integrations=[Integration.OPENAI]) > > client = OpenAI() > ``` > > Under a server such as uvicorn or gunicorn that block never runs, so initialize where the framework starts up. > [tab] TypeScript > > Initialize traceroot in the function your entry point calls to start, rather than at module scope, so a test that imports the module doesn't send traces. > > ```typescript theme={null} > // import 'dotenv/config'; // Uncomment if using a .env file (requires: npm install dotenv) > import OpenAI from 'openai'; > import { TraceRoot } from '@traceroot-ai/traceroot'; > > // This single line instruments all OpenAI API calls > TraceRoot.initialize({ instrumentModules: { openAI: OpenAI } }); > > const openai = new OpenAI(); > ``` > > If a framework loads your module instead of running it, initialize where the framework starts up instead. Run your own tests with `TRACEROOT_ENABLED=false` so they don't send traces either. For full SDK details, see the [Python SDK](/docs/tracing/python-sdk) or [TypeScript SDK](/docs/tracing/typescript-sdk). ### 5. See your first trace Run your application and exercise the code you instrumented. Open [traceroot Cloud](https://app.traceroot.ai) (or your self-hosted dashboard), navigate to your project, and you'll see the full span hierarchy, token usage, cost, and timing. Prefer the terminal? The [traceroot CLI](/docs/cli/get-started) works with the same traces, plus detector findings and evaluations, from your shell. ## Troubleshoot ### Setup stopped partway Ask setup what went wrong, then continue from where it stopped: ```bash theme={null} npx -y traceroot-cli@latest setup doctor npx -y traceroot-cli@latest setup --resume ``` If you ran setup with `--service `, pass the same option to both. If your coding agent didn't finish, setup keeps its output in `.traceroot/setup-instrument-output.log`. That file can quote your repository's files, secrets included, so check it before you share it. ### No trace arrived Run your application once and exercise the instrumented code, then run `npx -y traceroot-cli@latest setup --resume`. Neither SDK loads `.env` on its own, so check that your application loads the file setup wrote, or export the variables in your shell. ### Still stuck? Ask on [Discord](https://discord.gg/TM2m3CtKuC) or [open an issue](https://github.com/traceroot-ai/traceroot-cli/issues). ## Next steps > [card] [Python SDK](/docs/tracing/python-sdk) > > Full API reference for the Python SDK. > [card] [TypeScript SDK](/docs/tracing/typescript-sdk) > > Full API reference for the TypeScript SDK. > [card] [Features](/docs/tracing/overview) > > Attach users, sessions, tags, metadata, and more to your traces. > [card] [Detectors](/docs/detectors/introduction#how-to-create-a-detector) > > Create your first detector and start surfacing failures in your traces. > [card] [traceroot Agent](/docs/ai-agent/overview) > > Auto-debug failing traces with the traceroot Agent. > [card] [CLI](/docs/cli/get-started) > > Work with your traces, findings, and evaluations from the terminal. # Git Context Source: https://traceroot.ai/docs/tracing/git-context Correlate every span back to its source file, line, and commit traceroot automatically captures the git repository, commit, and the exact source file and line number for each instrumented function. This powers the **Show Code** feature in the trace viewer and enables the traceroot Agent to jump directly from a failing trace to the relevant code. ## What Gets Captured | Attribute | Description | | - | - | | `traceroot.git.repo` | Repository in `owner/repo` format | | `traceroot.git.ref` | Commit SHA, tag, or branch name | | `traceroot.git.source_file` | File path of the decorated function | | `traceroot.git.source_line` | Line number | | `traceroot.git.source_function` | Function name | `traceroot.git.repo` and `traceroot.git.ref` are set at the trace level. The `traceroot.git.source_*` attributes are set per span automatically by `@observe`. ## Configuration traceroot resolves the repository and ref independently, filling each unset value in this order: 1. Explicit initialization arguments. 2. `TRACEROOT_GIT_REPO` / `TRACEROOT_GIT_REF` environment variables. 3. GitHub Actions variables (`GITHUB_REPOSITORY` / `GITHUB_SHA`). 4. Files in `/.git` (`config` and `HEAD`). This step does not need the `git` binary and does not search parent directories. 5. A `git` subprocess (`git remote get-url origin` and `git rev-parse HEAD`). This step needs the binary but works when the process starts in a repository subdirectory. Automatic repository parsing recognizes only `github.com` remotes. For GitLab, Bitbucket, or self-hosted remotes, set `TRACEROOT_GIT_REPO` explicitly. Because the repository and ref resolve independently, one may be present while the other is missing. In that case the SDK logs `git context incomplete`; set the missing value explicitly so source correlation has both fields. Local `.git` auto-detection is a **development convenience only**. Production containers usually ship without a `.git` directory or the `git` binary, so auto-detection silently finds nothing and the AI agent cannot correlate traces to source. In production, always inject git context explicitly (see below). Set them explicitly via environment variables: ```bash theme={null} TRACEROOT_GIT_REPO=your-org/your-repo TRACEROOT_GIT_REF=9c1e4f2 ``` ## Git context in production Use **build-time injection**: inject the repo (static) and commit SHA (per-build) as environment variables at build time. The SDK reads them automatically — no code change required. ```dockerfile Dockerfile theme={null} ARG GIT_SHA ENV TRACEROOT_GIT_REPO=your-org/your-repo # static ENV TRACEROOT_GIT_REF=$GIT_SHA # dynamic, per build ``` ```yaml .github/workflows/deploy.yml theme={null} - name: Build image run: docker build --build-arg GIT_SHA=${{ github.sha }} -t myapp:${{ github.sha }} . ``` In **GitHub Actions**, traceroot reads the standard `GITHUB_REPOSITORY` and `GITHUB_SHA` automatically — no Dockerfile change needed. On other CI/platform providers, use the build-time injection above. Apply this to **every traced service**, including background workers and cron/queue consumers — not just your API server. Each process resolves git context independently. For setups that reuse one image across commits, set `TRACEROOT_GIT_REPO` / `TRACEROOT_GIT_REF` in the Kubernetes pod spec or ECS task definition at deploy time instead. ## Why It Matters Source code correlation is what enables the traceroot Agent to perform accurate root cause analysis — it can read the exact function that produced a failing span, not just the trace data. See [Root Cause Analysis](/docs/ai-agent/root-cause-analysis). # How AI agent tracing works Source: https://traceroot.ai/docs/tracing/introduction Follow model and tool calls through a run and see how many tokens each model call used. Tracing records what your AI agent does — every LLM call, tool invocation, and function execution — so you can debug individual runs and understand behavior over time. Once tracing is active, traceroot gives you: * **Full span hierarchy** — see exactly how your agent executes, call by call * **Token usage and cost** — per LLM call and aggregated per trace * **Source code correlation** — every span links back to the file and line that produced it ![traceroot trace viewer with a multi-step agent trace and an LLM span selected showing latency, tokens and cost](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/traces_v2.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=f32c9d8621afcd1da8ba84bbd6305516) Trace viewer: Trace Tree, LLM span details, Usage breakdown and Input ## Where to start > [card] [Get Started](/docs/tracing/get-started) > > Instrument your agent and send your first trace in minutes. > [card] [Features](/docs/tracing/overview) > > Attach users, sessions, tags, metadata, and more to your traces. # Live Tracing Source: https://traceroot.ai/docs/tracing/live-streaming Watch spans stream into the trace viewer in real time while your agent is still running When you open a trace whose root span hasn't ended yet, traceroot streams new spans into the viewer as they arrive. You don't need a toggle or an SDK flag — live tracing is on by default for every in-flight trace. ![Traces list with the Live toggle highlighted in the toolbar](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/trace_live_toggle_v1.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=76e6ccb5f8d5b36222a4d901f7549e7d) Traces list: Live toggle beside Columns and the date range The **Live** switch on the Traces list is separate: it refreshes the list every 5 seconds. ## How spans arrive Spans are merged into the existing tree by span ID, so they can arrive in any order: * Newly started spans appear as **pending** rows (greyed out, no duration yet). * When a pending span finishes, its end time fills in and the row renders normally. * The [Timeline View](/docs/tracing/timeline) updates in step — bars extend and new bars appear. Once the root span completes, the viewer makes a final consistency pass and stops streaming. # Metadata Source: https://traceroot.ai/docs/tracing/metadata Attach arbitrary key-value pairs to traces and spans **Metadata** is key-value pairs attached to traces or spans — useful for model versions, request IDs, feature flags, or any structured context you want to search on. ## Using `observe` Pass static metadata to `observe` for values known at definition time: > [tab] Python > > ```python theme={null} > from traceroot import observe > > @observe(name="process", type="agent", metadata={"version": "v2", "tier": "premium"}) > def process(query: str): > return result > ``` > [tab] TypeScript > > ```typescript theme={null} > import { observe } from '@traceroot-ai/traceroot'; > > const result = await observe( > { name: 'process', type: 'agent', metadata: { version: 'v2', tier: 'premium' } }, > async () => doWork(query), > ); > ``` ## Using `using_attributes` / `usingAttributes` Propagate metadata to all traces and spans within a block: > [tab] Python > > ```python theme={null} > from traceroot import using_attributes > > with using_attributes(metadata={"deploy": "canary", "region": "us-west-2"}): > agent.run(query) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { usingAttributes } from '@traceroot-ai/traceroot'; > > await usingAttributes({ metadata: { deploy: 'canary', region: 'us-west-2' } }, async () => { > await agent.run(query); > }); > ``` # Overview Source: https://traceroot.ai/docs/tracing/overview Attach context to your traces — users, sessions, tags, metadata, and more Once you have initialized traceroot, LLM calls are traced automatically. To go further — tracing custom functions, identifying users, grouping sessions, or tracking cost — you can use `observe` and attach rich context to every trace. ## Quickstart Here's everything you can attach in one place: > [tab] Python > > ```python theme={null} > import traceroot > from traceroot import Integration, observe, using_attributes > from openai import OpenAI > > traceroot.initialize(integrations=[Integration.OPENAI]) > > client = OpenAI() > > @observe(name="run_agent", type="agent", metadata={"version": "v2"}) > def run_agent(query: str) -> str: > response = client.chat.completions.create( > model="gpt-4o", > messages=[{"role": "user", "content": query}] > ) # tokens + cost tracked automatically > return response.choices[0].message.content > > with using_attributes(user_id="user-42", session_id="chat-abc", tags=["production"]): > run_agent("What's the weather?") > run_agent("What about tomorrow?") > ``` > [tab] TypeScript > > ```typescript theme={null} > import OpenAI from 'openai'; > import { TraceRoot, observe, usingAttributes } from '@traceroot-ai/traceroot'; > > TraceRoot.initialize({ instrumentModules: { openAI: OpenAI } }); > > const client = new OpenAI(); > > async function runAgent(query: string): Promise { > return observe({ name: 'run_agent', type: 'agent', metadata: { version: 'v2' } }, async () => { > const response = await client.chat.completions.create({ > model: 'gpt-4o', > messages: [{ role: 'user', content: query }] > }); // tokens + cost tracked automatically > return response.choices[0].message.content ?? ''; > }); > } > > await usingAttributes( > { userId: 'user-42', sessionId: 'chat-abc', tags: ['production'] }, > async () => { > await runAgent("What's the weather?"); > await runAgent('What about tomorrow?'); > }, > ); > ``` ## Quick Reference | I want to... | Attribute | Guide | | - | - | - | | Identify which user triggered a trace | User ID | [Users](/docs/tracing/users) | | Group traces from the same conversation | Session ID | [Sessions](/docs/tracing/sessions) | | Label traces for filtering | Tags | [Tags](/docs/tracing/tags) | | Attach structured key-value context | Metadata | [Metadata](/docs/tracing/metadata) | | Track token usage and LLM cost | auto via instrumentation | [Cost Tracking](/docs/tracing/cost-tracking) | | Link traces back to source code + commit | auto via `observe` | [Git Context](/docs/tracing/git-context) | # Python SDK Source: https://traceroot.ai/docs/tracing/python-sdk Installation, initialization, and full API reference for the traceroot Python SDK The traceroot Python SDK provides automatic and manual instrumentation for AI agents and LLM applications using OpenTelemetry. ## Installation Install the core SDK via `pip`: ```bash theme={null} pip install traceroot ``` ### Optional Extras Some integrations have additional dependencies and are available as extras: ```bash theme={null} # For DSPy integration pip install traceroot[dspy] # For Microsoft Agent Framework integration pip install traceroot[agent-framework] ``` Framework packages (such as `openai`, `anthropic`, or `langchain-core`) are installed separately alongside your application. ## Initialization Call `traceroot.initialize()` once at application startup to configure the SDK and enable auto-instrumentation for supported frameworks: ```python theme={null} import traceroot from traceroot import Integration traceroot.initialize( api_key="your-api-key", integrations=[ Integration.OPENAI, Integration.ANTHROPIC, Integration.LANGCHAIN, ], ) ``` > [!NOTE] > > **Auto-initialization**: The `@observe` decorator works without calling `initialize()`. If no client exists, `@observe` calls `get_client()`, which automatically initializes the SDK from environment variables (such as `TRACEROOT_API_KEY`). However, auto-initialization will not instrument any frameworks. To auto-instrument libraries, explicit `initialize(integrations=[...])` is required. > [!WARNING] > > **Repeat calls are ignored**: Calling `traceroot.initialize()` more than once logs a warning and returns the existing client instance without applying new parameters or integrations. Pass every setting you need on the first call; the SDK cannot be reconfigured once initialized. ### `initialize()` Parameters | Parameter | Type | Default | Env Var Fallback | Description | | - | - | - | - | - | | `api_key` | `str \| None` | `None` | `TRACEROOT_API_KEY` | traceroot API key for authentication. Required to export traces to the backend. | | `host_url` | `str \| None` | `"https://app.traceroot.ai"` | `TRACEROOT_HOST_URL` | traceroot backend API endpoint URL (or self-hosted instance URL). | | `flush_interval` | `float \| None` | `5.0` | `TRACEROOT_FLUSH_INTERVAL` | Interval in seconds between automatic background flushes. | | `batch_size` | `int \| None` | `100` | `TRACEROOT_FLUSH_AT` | Maximum number of spans to buffer before triggering a flush. Falls back to `TRACEROOT_FLUSH_AT`. | | `timeout` | `float \| None` | `30.0` | `TRACEROOT_TIMEOUT` | HTTP request timeout in seconds when exporting batches. | | `enabled` | `bool \| None` | `True` | `TRACEROOT_ENABLED` | Enable or disable tracing. If `False` or if `api_key` is not set, span exporting is disabled. | | `integrations` | `list[Integration] \| None` | `None` | — | List of `Integration` enum members to auto-instrument. | | `git_repo` | `str \| None` | auto-detected | `TRACEROOT_GIT_REPO` | Git repository in `owner/repo` format. Auto-detected from CI environment or local git repository if unset. | | `git_ref` | `str \| None` | auto-detected | `TRACEROOT_GIT_REF` | Git reference (commit SHA, tag, or branch). Auto-detected from CI environment or local git repository if unset. | | `redact` | `Callable[[dict[str, Any]], dict[str, Any]] \| None` | `None` | — | Rewrites each finished span's attributes before they are sent to traceroot. Other exporters in your app still receive the original spans. See [Redact personal data before export](/docs/tracing/data-capture#redact-personal-data-before-export). | ## Supported Integrations The `Integration` enum (`from traceroot import Integration`) provides auto-instrumentation for 17 AI frameworks and model providers. traceroot checks whether the required package is installed before instrumenting. If a package is not installed, the integration is skipped with a warning rather than raising an error. | Integration | Enum Member | Value | Required Package | Notes | | - | - | - | - | - | | OpenAI | `Integration.OPENAI` | `"openai"` | `openai` | Instruments OpenAI API client calls. | | Anthropic | `Integration.ANTHROPIC` | `"anthropic"` | `anthropic` | Instruments Anthropic API client calls. | | LangChain | `Integration.LANGCHAIN` | `"langchain"` | `langchain-core` or `langchain` | Instruments LangChain chains and LangGraph workflows. | | Google GenAI | `Integration.GOOGLE_GENAI` | `"google_genai"` | `google-genai` | Instruments Google Gemini / GenAI SDK. | | CrewAI | `Integration.CREWAI` | `"crewai"` | `crewai` | Instruments CrewAI agents and tasks. | | OpenAI Agents | `Integration.OPENAI_AGENTS` | `"openai_agents"` | `openai-agents` | Instruments OpenAI Agents SDK. | | Claude Agent SDK | `Integration.CLAUDE_AGENT_SDK` | `"claude_agent_sdk"` | `claude-agent-sdk` | Instruments Anthropic Claude Agent SDK via in-house wrapper. | | LlamaIndex | `Integration.LLAMA_INDEX` | `"llama_index"` | `llama-index-core` | Instruments LlamaIndex query engines, tools, and workflows. | | AutoGen | `Integration.AUTOGEN` | `"autogen"` | `ag2` | AutoGen instrumentation is gated on `ag2`. | | Agno | `Integration.AGNO` | `"agno"` | `agno` | Instruments Agno agents (formerly Phidata). | | Groq | `Integration.GROQ` | `"groq"` | `groq` | Instruments Groq API client calls. | | DSPy | `Integration.DSPY` | `"dspy"` | `dspy` or `dspy-ai` | Requires pip extra: `pip install traceroot[dspy]`. | | Google ADK | `Integration.GOOGLE_ADK` | `"google_adk"` | `google-adk` | Instruments Google Agent Development Kit. | | Mistral | `Integration.MISTRAL` | `"mistral"` | `mistralai` | Instruments Mistral AI client calls. | | PydanticAI | `Integration.PYDANTIC_AI` | `"pydantic_ai"` | `pydantic-ai` or `pydantic-ai-slim` | Instruments PydanticAI agents and models. | | AWS Bedrock | `Integration.BEDROCK` | `"bedrock"` | `boto3` | Instruments Amazon Bedrock runtime calls via `boto3`. | | Agent Framework | `Integration.AGENT_FRAMEWORK` | `"agent_framework"` | `agent-framework-core` | Requires pip extra: `pip install traceroot[agent-framework]`. | ## The Observe Decorator Wrap any function to create an OpenTelemetry span. Supports synchronous functions, asynchronous functions, and generators. ```python theme={null} from traceroot import observe @observe(name="my_function", type="tool") def my_function(query: str) -> str: return f"Processed: {query}" @observe(name="async_agent", type="agent") async def async_agent(query: str) -> str: return f"Result: {query}" ``` ### Parameters | Parameter | Type | Default | Description | | - | - | - | - | | `name` | `str \| None` | function name | Custom span name. Defaults to the decorated function's name. | | `type` | `SpanKind \| str` | `"span"` | Span kind: `"llm"`, `"agent"`, `"tool"`, or `"span"`. Accepts string literals or `SpanKind` imported via `from traceroot.constants import SpanKind`. Unrecognized values emit a warning and fall back to `"span"`. (The enum also defines `"evaluation"`, `"task"`, and `"scorer"` used internally by evaluations). | | `metadata` | `dict[str, Any] \| None` | `None` | Static metadata dictionary to attach to the span. | | `tags` | `list[str] \| None` | `None` | Tags to categorize the span. | | `capture_input` | `bool` | `True` | Whether to auto-capture function arguments as input. | | `capture_output` | `bool` | `True` | Whether to auto-capture the return value as output. | | `session_id` | `str \| None` | `None` | Session identifier to attach to the trace. | | `user_id` | `str \| None` | `None` | User identifier to attach to the trace. | See [Data Capture and Privacy](/docs/tracing/data-capture) for examples that keep sensitive function inputs and outputs off spans. ## Setting Trace Context Use `using_attributes` to set user and session context for all spans created within a block. This is re-exported from OpenInference for convenience: ```python theme={null} from traceroot import using_attributes with using_attributes( user_id="user-123", session_id="sess-456", tags=["production", "v2"], metadata={"feature": "chat"}, ): result = run_agent("What is the weather?") ``` ### Parameters | Parameter | Type | Description | | - | - | - | | `user_id` | `str` | User identifier. | | `session_id` | `str` | Session / conversation ID. | | `tags` | `list[str]` | Tags to attach to all spans created within the context. | | `metadata` | `dict` | Arbitrary metadata dictionary. | ## Updating Spans and Traces Update the active span or trace programmatically from within any traced function: ```python theme={null} from traceroot import update_current_span, update_current_trace # Update the current span update_current_span( name="custom_name", input={"query": "hello"}, output={"response": "world"}, metadata={"custom_key": "custom_value"}, # LLM-specific attributes (useful for custom/unsupported providers) model="gpt-4o", model_parameters={"temperature": 0.7, "max_tokens": 1024}, usage={"input_tokens": 100, "output_tokens": 50}, prompt=[{"role": "user", "content": "hello"}], ) # Update the current trace update_current_trace( user_id="user-123", session_id="sess-456", tags=["production", "v2"], metadata={"plan": "pro", "region": "us-east-1"}, ) ``` > [!NOTE] > > Both `update_current_span` and `update_current_trace` accept **keyword-only arguments**. Calling them with positional arguments will raise a `TypeError`. ### `update_current_span` Parameters | Parameter | Type | Description | | - | - | - | | `name` | `str \| None` | Update the active span name. | | `input` | `Any` | Input data (e.g. messages or arguments sent to an LLM). | | `output` | `Any` | Output data (e.g. LLM completion or function return value). | | `metadata` | `dict[str, Any] \| None` | Custom metadata key-value pairs. | | `model` | `str \| None` | LLM model name (e.g. `"gpt-4o"`, `"claude-3-5-sonnet"`). | | `model_parameters` | `dict[str, Any] \| None` | Model parameters (e.g. `{"temperature": 0.7, "max_tokens": 1024}`). | | `usage` | `dict[str, int] \| None` | Token usage dict (e.g. `{"input_tokens": 100, "output_tokens": 50}`). | | `cost` | `float \| None` | Total cost of this call in USD, for an LLM call or a paid tool or API call. Replaces traceroot's computed price for the span. | | `prompt` | `Any` | Prompt or messages payload sent to the model. | ### `update_current_trace` Parameters | Parameter | Type | Description | | - | - | - | | `user_id` | `str \| None` | User identifier who initiated the trace. | | `session_id` | `str \| None` | Session identifier for grouping related traces into a conversation. | | `metadata` | `dict[str, Any] \| None` | Trace-level metadata key-value pairs. | | `tags` | `list[str] \| None` | Tags to categorize and filter the trace in the UI. | ## SDK Utilities and Context The `traceroot` package exports client management and OpenTelemetry context utilities: ```python theme={null} import traceroot from traceroot import ( get_client, flush, shutdown, get_current_trace_id, get_current_span_id, ) @traceroot.observe() def my_function(): # Retrieve current active OpenTelemetry IDs trace_id = get_current_trace_id() span_id = get_current_span_id() print(f"Trace ID: {trace_id}, Span ID: {span_id}") # Flush pending spans (e.g. in serverless functions or notebooks) traceroot.flush() # Graceful shutdown on application exit traceroot.shutdown() ``` ### Exported Utilities | Symbol | Type | Description | | - | - | - | | `flush()` | Function | Flushes all pending spans and traces immediately to the backend. Crucial in serverless environments (AWS Lambda, Google Cloud Functions) and Jupyter notebooks before process termination. | | `shutdown()` | Function | Flushes pending spans and stops background exporter worker threads gracefully. Call it once, on application exit. | | `get_client()` | Function | Returns the singleton `TracerootClient` instance, auto-initializing one from environment variables if none exists. | | `TracerootClient` | Class | The client class instantiated by `initialize()` and `get_client()`. Manages batching, background workers, and OpenTelemetry trace exporting. | | `get_current_trace_id()` | Function | Returns the active 32-character hexadecimal trace ID string (e.g. `"4bf92f3577b34da6a3ce929d0e0e4736"`), or `None` if there is no active recording span. | | `get_current_span_id()` | Function | Returns the active 16-character hexadecimal span ID string (e.g. `"00f067aa0ba902b7"`), or `None` if there is no active recording span. | ## Offline Evaluations traceroot includes a complete offline evaluation framework exported directly from the top-level `traceroot` module: * **Datasets & Cases**: `Dataset`, `DatasetSnapshot`, `EvalCase`, `pull_dataset`, `pull_dataset_version` * **Execution**: `evaluate`, `evaluate_async`, `Evaluation`, `EvalRunResult`, `EvalItemResult`, `RunDatasetRef` * **Scorers & Metrics**: `Scorer`, `ScorerContext`, `scorer`, `Score`, `ScoreSummary`, `llm_judge`, `DeferredScore` * **Exceptions**: `DatasetConflictError`, `DatasetPublishAborted`, `EvalCompletionError` Evaluation symbols are loaded lazily on demand to keep imports lightweight. > [card] [Evaluations Overview](/docs/evals/introduction) > > Learn how offline evaluations work in traceroot. > [card] [Running Evals](/docs/evals/running-evals) > > Run your AI system against datasets and score outputs with `evaluate()`. ## Environment Variables All parameters except `integrations` can be configured via environment variables: | Variable | Default | Matches Parameter | Description | | - | - | - | - | | `TRACEROOT_API_KEY` | (required) | `api_key` | traceroot API key for authentication. | | `TRACEROOT_HOST_URL` | `https://app.traceroot.ai` | `host_url` | API endpoint URL. | | `TRACEROOT_ENABLED` | `true` | `enabled` | Enable/disable tracing. | | `TRACEROOT_FLUSH_INTERVAL` | `5.0` | `flush_interval` | Interval in seconds between automatic flushes. | | `TRACEROOT_FLUSH_AT` | `100` | `batch_size` | Batch size before flush (note: maps to `batch_size` in `initialize()`). | | `TRACEROOT_TIMEOUT` | `30.0` | `timeout` | HTTP request timeout in seconds. | | `TRACEROOT_GIT_REPO` | auto-detected | `git_repo` | Git repository (`owner/repo`). | | `TRACEROOT_GIT_REF` | auto-detected | `git_ref` | Git reference (commit SHA, tag, or branch). | # Sessions Source: https://traceroot.ai/docs/tracing/sessions Group related traces into sessions for multi-turn conversations and workflows Set a **Session ID** to group traces together — all turns in a conversation, or all steps in a multi-agent workflow. ![traceroot Session panel with session stats, user, and each trace's input and output](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/session_v2.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=17311bbc1eb6c6585b23fee13ad5509f) Session panel: summary stats, user, then each trace's input and output ## Using `using_attributes` / `usingAttributes` Propagate a Session ID to all traces within a block: > [tab] Python > > ```python theme={null} > from traceroot import using_attributes > > with using_attributes(session_id="chat-abc-123"): > response_1 = agent.run("What's the weather in SF?") > response_2 = agent.run("What about tomorrow?") > ``` > > Combine with a User ID to get full context on every trace: > > ```python theme={null} > with using_attributes(session_id="chat-abc-123", user_id="user-42"): > agent.run(query) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { usingAttributes } from '@traceroot-ai/traceroot'; > > await usingAttributes({ sessionId: 'chat-abc-123' }, async () => { > await agent.run("What's the weather in SF?"); > await agent.run('What about tomorrow?'); > }); > ``` > > Combine with a User ID to get full context on every trace: > > ```typescript theme={null} > await usingAttributes({ sessionId: 'chat-abc-123', userId: 'user-42' }, async () => { > await agent.run(query); > }); > ``` ## Using `observe` / inside a function Call `update_current_trace` / `updateCurrentTrace` from within any observed function: > [tab] Python > > ```python theme={null} > from traceroot import observe, update_current_trace > > @observe(name="handle_request", type="agent") > def handle_request(query: str, session_id: str): > update_current_trace(session_id=session_id) > return process(query) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { observe, updateCurrentTrace } from '@traceroot-ai/traceroot'; > > async function handleRequest(query: string, sessionId: string) { > return observe({ name: 'handle_request', type: 'agent' }, async () => { > updateCurrentTrace({ sessionId }); > return handleQuery(query); > }); > } > ``` # Tags Source: https://traceroot.ai/docs/tracing/tags Label traces with string tags for filtering and categorization **Tags** are string labels on traces for filtering and categorization — by environment, experiment, feature, or any dimension you care about. ## Using `observe` Pass static tags directly to `observe`: > [tab] Python > > ```python theme={null} > from traceroot import observe > > @observe(name="run_agent", type="agent", tags=["production", "v2"]) > def run_agent(query: str): > return process(query) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { observe } from '@traceroot-ai/traceroot'; > > const result = await observe( > { name: 'run_agent', type: 'agent', tags: ['production', 'v2'] }, > async () => handleQuery(query), > ); > ``` ## Using `using_attributes` / `usingAttributes` Propagate tags to all traces within a block: > [tab] Python > > ```python theme={null} > from traceroot import using_attributes > > with using_attributes(tags=["staging", "experiment-a"]): > agent.run(query) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { usingAttributes } from '@traceroot-ai/traceroot'; > > await usingAttributes({ tags: ['staging', 'experiment-a'] }, async () => { > await agent.run(query); > }); > ``` # Timeline View Source: https://traceroot.ai/docs/tracing/timeline Visualize spans as Gantt-style bars over time to spot long operations, parallelism, and gaps The trace viewer has three views: **Trace** (hierarchical tree), **Timeline** (Gantt-style bars), and **Detectors** (findings produced for the trace). The Trace and Timeline views share selection and collapse state. ![traceroot trace viewer in Timeline view with the span tree and duration bars on a time axis](https://mintcdn.com/tracerootai/TLv21OVXbJzkJk5L/images/trace_timeline_v2.png?fit=max\&auto=format\&n=TLv21OVXbJzkJk5L\&q=85\&s=128c277233f7f242c62ae91f46d3bff1) Timeline view: view toggle, span tree, duration bars on a time axis ## Trace, Timeline, and Detectors | Trace view | Timeline view | Detectors view | | - | - | - | | Call hierarchy and parent/child relationships | Span duration relative to siblings | Findings produced by detectors | | Inspecting span attributes, inputs, outputs | Finding the critical path or longest span | Reviewing detector results for this trace | | Navigating nested agent or tool calls | Checking parallel vs serial execution | Opening the related analysis | The Detectors view is hidden for detector and agent traces, which detectors do not evaluate, and when the trace viewer is embedded in an evaluation view. ## Toggling Use the **Trace**, **Timeline**, and **Detectors** buttons in the trace viewer's sub-header. Selection persists when switching between Trace and Timeline. ## Click-to-select Clicking a bar in the Timeline view selects the span and switches back to the Trace view, scrolling and expanding ancestors so the span is visible. # TypeScript SDK Source: https://traceroot.ai/docs/tracing/typescript-sdk Full API reference for the traceroot TypeScript SDK The traceroot TypeScript SDK provides automatic and manual instrumentation for AI agents using OpenTelemetry. ## Instrumentation Pass module instances to `initialize()` to instrument all calls automatically: ```typescript theme={null} import OpenAI from 'openai'; import { TraceRoot } from '@traceroot-ai/traceroot'; TraceRoot.initialize({ instrumentModules: { openAI: OpenAI }, }); const openai = new OpenAI(); ``` ## The Observe Wrapper Wrap any async function to create a span: ```typescript theme={null} import { observe } from '@traceroot-ai/traceroot'; const result = await observe({ name: 'my_function', type: 'tool' }, async () => { return 'your result here'; }); ``` ### Parameters | Parameter | Type | Default | Description | | - | - | - | - | | `name` | `string` | function name | Span name | | `type` | `'span' \| 'agent' \| 'tool' \| 'llm'` | `'span'` | Span kind — `'agent'` for multi-step agents, `'tool'` for function calls, `'llm'` for model inference, `'span'` for generic | | `metadata` | `Record` | — | Static metadata to attach | | `tags` | `string[]` | — | Tags to attach | | `captureInput` | `boolean` | `true` | Capture the function arguments as span input | | `captureOutput` | `boolean` | `true` | Capture the return value as span output | See [Data Capture and Privacy](/docs/tracing/data-capture) for `captureInput` and `captureOutput` examples that keep sensitive values off spans. To rewrite personal data in every span before export, pass a `redact` function to `TraceRoot.initialize()`. It may be async. See [Data Capture and Privacy](/docs/tracing/data-capture#redact-personal-data-before-export). ## Setting Trace Context Use `usingAttributes()` to set user and session context for all spans created within a block: ```typescript theme={null} import { usingAttributes } from '@traceroot-ai/traceroot'; await usingAttributes( { sessionId: 'conv-123', userId: 'user-456', tags: ['production', 'v2'], metadata: { feature: 'chat' }, }, async () => { // All spans here inherit the attributes above await runAgent('What is the weather?'); }, ); ``` ### Parameters | Parameter | Type | Description | | - | - | - | | `sessionId` | `string` | Session / conversation ID | | `userId` | `string` | User identifier | | `tags` | `string[]` | Tags to attach to all spans | | `metadata` | `Record` | Arbitrary metadata | ## Updating Spans and Traces Update the current span or trace programmatically: ```typescript theme={null} import { observe, updateCurrentSpan, updateCurrentTrace } from '@traceroot-ai/traceroot'; await observe({ name: 'my_agent', type: 'agent' }, async () => { updateCurrentSpan({ input: { query: userInput }, output: { response: agentOutput }, model: 'gpt-4o', usage: { input: 100, output: 50 }, }); updateCurrentTrace({ userId: 'user-123', sessionId: 'sess-456', tags: ['prod'], metadata: { plan: 'pro' }, }); }); ``` ### `updateCurrentSpan` Parameters | Parameter | Type | Description | | - | - | - | | `name` | `string` | Rename the span after creation | | `input` | `unknown` | Input data | | `output` | `unknown` | Output data | | `metadata` | `Record` | Custom metadata | | `model` | `string` | LLM model name | | `modelParameters` | `Record` | LLM model parameters | | `usage` | `TokenUsage` | Token usage (`{ input, output, cacheRead, cacheWrite, total }`) | | `cost` | `number` | Total cost of this call in USD, for an LLM call or a paid tool or API call. Replaces traceroot's computed price for the span | | `prompt` | `unknown` | Prompt or messages sent to the LLM | | `attributes` | `Record` | Arbitrary span attributes | ### `updateCurrentTrace` Parameters | Parameter | Type | Description | | - | - | - | | `userId` | `string` | User identifier | | `sessionId` | `string` | Session identifier | | `tags` | `string[]` | Tags for the trace | | `metadata` | `Record` | Trace-level metadata | ## Flushing and Shutdown By default, the TypeScript SDK batches spans to optimize performance, flushing them every 5 seconds or when 100 spans are accumulated. **Important:** The SDK's only automatic shutdown flush is a `beforeExit` hook. Node.js does not emit `beforeExit` on `process.exit()`, unhandled exceptions, or signals like SIGINT/SIGTERM. Serverless runtimes that freeze the process (e.g. AWS Lambda, Cloud Functions) skip it too. In any of these cases, buffered spans will be lost if you exit without flushing explicitly. To ensure all spans are exported before your script terminates, await `shutdown()` (or `flush()`): ```typescript theme={null} import { TraceRoot } from '@traceroot-ai/traceroot'; TraceRoot.initialize(); async function main() { try { // ... your application logic ... } finally { // Ensure all batched spans are exported before exiting, even on failure await TraceRoot.shutdown(); } } main(); ``` Alternatively, for tests or short scripts, you can disable batching: ```typescript theme={null} TraceRoot.initialize({ disableBatch: true }); ``` **Note:** Disabling batching does not make the network request synchronous. You still must await `flush()` or `shutdown()` before terminating the process with `process.exit()`. *(Note: The Python SDK registers an `atexit` handler that flushes spans automatically in most environments. However, `atexit` does not run when a serverless runtime freezes the process — Python handlers there still need `traceroot.flush()` before returning.)* ## Environment Variables | Variable | Default | Description | | - | - | - | | `TRACEROOT_API_KEY` | (required) | API key | | `TRACEROOT_HOST_URL` | `https://app.traceroot.ai` | API endpoint | | `TRACEROOT_ENABLED` | `true` | Enable/disable tracing | | `TRACEROOT_ENVIRONMENT` | — | Deployment environment tag (e.g. `production`, `staging`) | | `TRACEROOT_GIT_REPO` | auto-detected | Git repository (`owner/repo`) | | `TRACEROOT_GIT_REF` | auto-detected | Git reference | # Users Source: https://traceroot.ai/docs/tracing/users Associate traces with specific users Set a **User ID** on a trace to associate it with a specific user — enabling per-user filtering, cost tracking, and analytics in the dashboard. ## Using `using_attributes` / `usingAttributes` Propagate a User ID to all traces within a block: > [tab] Python > > ```python theme={null} > from traceroot import using_attributes > > with using_attributes(user_id="user-42"): > response = agent.run("What's the weather in SF?") > ``` > [tab] TypeScript > > ```typescript theme={null} > import { usingAttributes } from '@traceroot-ai/traceroot'; > > await usingAttributes({ userId: 'user-42' }, async () => { > await agent.run("What's the weather in SF?"); > }); > ``` ## Using `observe` / inside a function Call `update_current_trace` / `updateCurrentTrace` from within any observed function: > [tab] Python > > ```python theme={null} > from traceroot import observe, update_current_trace > > @observe(name="handle_request", type="agent") > def handle_request(query: str, user_id: str): > update_current_trace(user_id=user_id) > return process(query) > ``` > [tab] TypeScript > > ```typescript theme={null} > import { observe, updateCurrentTrace } from '@traceroot-ai/traceroot'; > > async function handleRequest(query: string, userId: string) { > return observe({ name: 'handle_request', type: 'agent' }, async () => { > updateCurrentTrace({ userId }); > return handleQuery(query); > }); > } > ```