Skip to main content
Tracing records what happened inside every call — the model call, any nested tool or LLM calls, and every Control Plane rule that fired. Each call becomes a trace: a tree of spans you can open and inspect. A typical use: figure out why one request was slow or wrong — which model ran, how long each step took, what a tool returned, and what a judge scored it.
By default, traces record metadata only — model, cost, and latency. To capture full inputs and outputs, turn on retention with a Comply rule.

​
What you see in the dashboard

Open Traces in platform.opper.ai. Three views build on each other. The trace explorer lists recent traces, each row showing the time, total duration, the trace name with its step composition (e.g. 28 steps · 8 llm · 11 tool), cost and tokens, and a preview of the input and output. A status dot flags failures, and you can search by trace name or filter by Status, Duration, and Cost.
The Traces explorer listing recent traces with duration, step composition, cost, tokens, and an input/output preview
The span tree opens when you click a trace. The left pane lists every span as a row with its name and duration, drawn as a timeline bar so you can see where time went and which calls fanned out from which. Generations (llm) and tool calls (web_search, web_fetch, …) nest under the step that ran them. The header sums up the whole trace — duration, tokens, and cost.
A trace's span tree with nested LLM and tool-call spans, each showing its duration, beside the selected span's input and output
The span detail pane shows everything for the selected span: its span ID, parent, where it starts within the trace, its duration, and — with retention on — the full input and output (rendered as chat, or as raw JSON with the {} toggle). Click into any span to drill down through the breadcrumb at the top.
The span detail pane for a web_fetch span showing span ID, parent, start offset, input, and output

​
Quick actions

A few actions sit in the header of each view:
ActionWhat it does
Open in playgroundLoads the trace into the playground as an editable conversation so you can replay and iterate on it (see below).
Copy spanCopies the span’s full JSON (input, output, metadata, timings) to your clipboard, handy for sharing a repro or pasting into an issue.
Open trace in new tabOpens the full trace on its own page (via the ↗ icon), so you can keep it open while you work or share a direct link to it.

​
Replay a trace in the playground

Open in playground reconstructs the trace as an editable conversation — every system, user, and assistant turn, in order — so you can pick up a real production run and iterate on it.
A trace loaded into the playground as a conversation, with a Run from here button on a message and a model switcher in the side panel
From here you can:
  • Run from here — hover any message and replay the conversation from that point, so you can change one turn and see how the run plays out without rerunning everything before it.
  • Switch the model — pick a different model from the Model dropdown to see how another model handles the same input, and adjust Parameters (temperature, max tokens), Controls, and Tools alongside it.
  • View code, Compare, or Save the result — none of this touches production, so it’s a safe place to debug a bad trace or tune a prompt.

​
The trace model

A trace is a tree of spans. The root span names the trace; every other span points to its parent through parent_id, which is how the tree is built. Spans of type generation are the LLM calls; tool and function calls appear as their own spans nested under the step that invoked them.
FieldWhat it holds
nameHuman-readable label for the step. The root span’s name becomes the trace name.
typeSpan kind — e.g. generation (an LLM call), function, or call.
input / outputThe data into and out of the step. Stored only with retention on.
start_time / end_timeUTC timestamps; the dashboard derives a duration from them.
parent_idThe parent span, used to build the tree. Empty on the root span.
errorError message if the step failed.
meta / tagsArbitrary metadata you attach for filtering and context.
scoreNumeric score on the span.
metricsCustom measurements, each a dimension + value (+ optional comment).
Generation spans carry extra detail — the model that ran, total_tokens, the instructions used, and, when Observe runs, the judge’s observations and per-criterion scorer_context. The trace itself rolls up its spans: name, input, output, start_time/end_time, duration_ms, status, span_count, and total_tokens, plus the ordered spans tree and any events.

​
Retention

Retention rules attach at the org or project level, so you can set different policies for different projects — full retention on a staging project while you debug, metadata only on a sensitive production one, and so on. Set them with a Comply rule. See Core concepts for how this fits into the request path.

​
Instrument custom spans

Most spans are created for you. To trace work that happens outside a single Gateway call — a multi-step pipeline, a background job, your own retrieval step — create spans yourself. A span with no parent_id starts a new trace; pass parent_id to nest a child under it. 
from datetime import datetime, timezone
from opperai import Opper

opper = Opper()

# Root span — starts a new trace
span = opper.spans.create(
    name="my-pipeline",
    start_time=datetime.now(timezone.utc).isoformat(),
    input="Starting the pipeline",
    meta={"userId": "u-123"},
)

# Child span — nested under the root via parent_id
step = opper.spans.create(
    name="retrieve-context",
    parent_id=span.id,
    start_time=datetime.now(timezone.utc).isoformat(),
)

# Close a span with its output and end time
opper.spans.update(
    step.id,
    output="Found 5 documents",
    end_time=datetime.now(timezone.utc).isoformat(),
)

print(f"Span: {span.id}, Trace: {span.trace_id}")

​
Read traces programmatically

List recent traces, then fetch one to walk its span tree. 
from opperai import Opper

opper = Opper()

traces = opper.traces.list(limit=5)
for t in traces.data:
    print(f"{t.id} - {t.name or '(unnamed)'} ({t.span_count} spans)")

trace = opper.traces.get(traces.data[0].id)
for s in trace.spans:
    indent = "  " if not s.parent_id else "    "
    print(f"{indent}{s.name} ({s.id[:8]}...)")

​
Events and feedback on spans

As a call runs, the Control Plane records what it did as events on the relevant span. Each event has a kind — guardrail, observe, route, comply, or feedback — a source, and a data payload. You don’t create these; turning on a rule does. They’re what populate the Controls section of the span detail. You can also write your own events — for example, capturing thumbs-up/down feedback from your app — with POST /v3/spans/{id}/events, and read them back with GET /v3/spans/{id}/events.
Use meta and tags on your custom spans to record context like user, tenant, or feature flag. It makes traces far easier to filter and group later.

​
Where to go next

Observe

Score every response against criteria you write. Results land on the trace.

Comply

Turn on retention so traces store full inputs and outputs.

Spans API

Create, update, and read spans directly over the API.

Core concepts

How traces fit into the two-plane model and the request path.