Tracing

What is Tracing?

Tracing is essentially logging, but where related or subsequent logs is associated with eachother so it is possible to look at them as a whole.

Why Tracing?

This drastically simplifies debugging and understanding complex interactions in chatbot sessions, agent runs or similar.

With the Opper Python and Typescript SDK, you can:

Connect related operations together in a trace to see the full flow of a session, agent run or chain.
Store feedback, results from evaluations and other data relevant for understanding the quality of the session with metrics.
Add contextual information such as user id, session id, agent id etc as metadata.

Calls automatically create spans

When doing calls, the SDK will automatically create a span:

from opperai import AsyncOpper
import asyncio

opper = AsyncOpper()

async def main():
    question = "What is the capital of Sweden?"

    result, response = await opper.call(
        name="answer",
        instructions="Answer the question",
        input=question,
    )

    span = response.span

    print(f"Span: {span}")
    # Span: AsyncSpan(_client=<opperai._client.AsyncClient object at 0x7f8a56e7a2d0>, _uuid='8846d145-0e40-406a-8d5f-4f817fe864a9')

asyncio.run(main())

import Client from "opperai";

const client = new Client();

async function main() {
    const question = "What is the capital of Sweden?";

    const { message, span_id } = await client.call({
        name: "answer",
        instructions: "Answer the question",
        input: question,
    });

    console.log(`Span UUID: ${span_id}`);
    // Span UUID: 123e4567-e89b-12d3-a456-426614174000
}

main();

Adding a Span to a Trace

In the following example we will connect a call span to a higher level span.

from opperai import AsyncOpper
import asyncio

opper = AsyncOpper()

async def main():

    question = "What is the capital of Sweden?"

    # Create a simple span for tracing
    # All calls and spans below in this context manager will be connected to this span
    async with opper.traces.start(
        name="session",
        input=question) as span:
        
        # Calls automatically creates its own span and connects to the parent span
        result, response = await opper.call(
            name="answer",
            instructions="Answer the question",
            input=question,
        )
        
        # We update the higher level span with the final output
        await span.update(output=result)

asyncio.run(main())

from opperai import AsyncOpper, trace
import asyncio

opper = AsyncOpper()

# We use the decorator to trace the function
# With trace_io=True, we add the arguments as span input and return value as output
@trace(name="my_span", trace_io=True)
async def main(question):

    # The call automatically creates its own span and connects to the parent span
    result, response = await opper.call(
        name="answer",
        instructions="Answer the question",
        input=question,
    )
    
    # if not using trace_io, you can update the current span with the output like this:
    # await opper.traces.current_span.update(output=result)
    return result

asyncio.run(main(question="What is the capital of Sweden?"))

import Client from "opperai";

const client = new Client();

async function main() {

    const question = "What is the capital of Sweden?";

    // We create a parent span for the session
    const parentSpan = await client.traces.start({
        name: "session",
        input: question,
    });

    try {
        // The call automatically creates its own span
        // We just need to connect it to the parent span
        const { message } = await client.call({
            name: "answer",
            instructions: "Answer the question",
            input: question,
            parent_span_uuid: parentSpan.uuid, // Connect to parent span
        });

        // End the parent span and add the call output as output to the parent span
        await parentSpan.end({ output: message });
    } catch (error) {
        console.error("Error:", error);
    }
}

main();

Adding Metrics to a Span

You can also add metrics to spans. This is useful for evaluating the quality of the session, agent run or similar.

In the following example we will evaluate if the answer is one word and save the verdict as a metric on the span.

from opperai import AsyncOpper
import asyncio

opper = AsyncOpper()


async def main():
    question = "What is the capital of Sweden?"

    # call the model to get an answer
    # the call automatically creates a span
    result, response = await opper.call(
        name="one_word_answer",
        instructions="Answer the question in one word",
        input=question,
    )

    # get the span_uuid from the response
    await response.span.save_metric(
        dimension= "eval_correctness",  # metric name 
        value = 1 if len(result.split()) == 1 else 0,  # metric value
        comment = f"We evaluated that the answer is one word"  # comment
    )

asyncio.run(main())

from opperai import AsyncOpper, trace
import asyncio

opper = AsyncOpper()

@trace(name="example", trace_io=True)
async def main():

    # get the span_uuid from the response
    span = opper.traces.current_span

    # save the metric on the span
    await span.save_metric(
        dimension= "some_dimension",  # metric name 
        value = 1,  # metric value
        comment = "Some comment"  # comment
    )

asyncio.run(main())

import Client from "opperai";

const client = new Client();

async function main() {
    // call the model to get an answer
    const { message, span_id } = await client.call({
        name: "one_word_answer",
        instructions: "Answer the question in one word",
        input: "What is the capital of Sweden?",
    });

    // save a metric on the span
    await client.spans.saveMetric(span_id, {
        dimension: "eval_correctness", // metric name
        value: 1 if message.split(" ").length === 1 else 0, // metric value
        comment: "Is the answer one word? true/false", // comment
    });
}

main();

Investigate Traces

Traces show up in the portal https://platform.opper.ai/traces. There you can see all traces for your project, including token use, evaluations etc.

Viewing a trace

Click on a trace to view all calls in a trace. Under each call is a generation where you can see the prompt, model used etc.

More information

See more extensive examples in our Python and Node.js SDKs, who both have an example folder with runnable examples.