Evaluations

Evaluation is a critical part of building reliable LLM applications. It helps you verify your function behaves correctly across different inputs, compare performance between different models and prompts, track quality over time as you make changes, and build confidence in your LLM features before or during deployment.

Opper provides two types of evaluation:

Online Evaluation: Automatically runs on every function call in real-time, providing immediate feedback on output quality and correctness.
Offline Evaluation: Runs against your curated datasets, allowing you to systematically test your function against known test cases and ensure consistent behavior across important scenarios.

Together, these evaluation methods help you maintain high quality standards - catching issues as they happen while also verifying reliability across critical test cases.

Using Evaluation in Your Code

The Opper SDKs include support for creating custom evaluators and running evaluations directly in your code. This enables you to programmatically evaluate your LLM outputs using metrics tailored to your specific use case.

Creating Custom Evaluators

Custom evaluators are functions that take an LLM result and return metrics. Here's a simple example:

from opperai import AsyncOpper, evaluator
from opperai.types import Metric

opper = AsyncOpper()

@evaluator
def word_count_evaluator(result, min_words=5, max_words=20):
    """Evaluator that checks if the text has an appropriate number of words."""
    # Count words
    words = str(result).split()
    word_count = len(words)
    
    # Calculate score (1.0 is perfect)
    if min_words <= word_count <= max_words:
        score = 1.0
    elif word_count < min_words:
        score = word_count / min_words
    else:
        score = max(0, 1 - (word_count - max_words) / max_words)
        
    return [
        Metric(
            dimension="word_count", 
            value=score,
            comment=f"Found {word_count} words (expected {min_words}-{max_words})"
        )
    ]

import { Client, evaluator, Metric } from "opperai";

const client = new Client();

const wordCountEvaluator = evaluator(
  async (
    result: string,
    minWords: number = 5,
    maxWords: number = 20
  ): Promise<Metric[]> => {
    // Count words
    const words = result.split(/\s+/);
    const wordCount = words.length;
    
    // Calculate score (1.0 is perfect)
    let score: number;
    if (minWords <= wordCount && wordCount <= maxWords) {
      score = 1.0;
    } else if (wordCount < minWords) {
      score = wordCount / minWords;
    } else {
      score = Math.max(0, 1 - (wordCount - maxWords) / maxWords);
    }
    
    return [
      {
        dimension: "word_count",
        value: score,
        comment: `Found ${wordCount} words (expected ${minWords}-${maxWords})`,
      } as Metric
    ];
  }
);

Creating LLM-based Evaluators

You can also create evaluators that use LLM calls for more subjective evaluations:

from pydantic import BaseModel, Field
from typing import Literal

@evaluator
async def tone_evaluator(result, target_tone="friendly"):
    """Evaluator that checks if the text matches a target tone."""
    class ToneAnalysis(BaseModel):
        matches_tone: bool = Field(description="Whether the text matches the target tone")
        confidence: float = Field(description="Confidence score (0-1)", ge=0.0, le=1.0)

    # Ask LLM to evaluate the tone
    response, _ = await opper.call(
        name="tone_analysis",
        instructions=f"Evaluate if the text has a {target_tone} tone.",
        input=result,
        output_type=ToneAnalysis
    )
    
    return [
        Metric(
            dimension="tone.match",
            value=1.0 if response.matches_tone else 0.0,
            comment=f"Target tone: {target_tone}"
        ),
        Metric(
            dimension="tone.confidence",
            value=response.confidence,
            comment="Confidence in tone assessment"
        )
    ]

import { Client, evaluator, Metric } from "opperai";

const client = new Client();

const toneEvaluator = evaluator(
  async (
    result: string,
    targetTone: string = "friendly"
  ): Promise<Metric[]> => {
    // Ask LLM to evaluate the tone
    const { json_payload } = await client.call({
      name: "tone_analysis",
      instructions: `Evaluate if the text has a ${targetTone} tone.`,
      input: result,
      output_schema: {
        type: "object",
        properties: {
          matches_tone: {
            type: "boolean",
            description: "Whether the text matches the target tone"
          },
          confidence: {
            type: "number",
            minimum: 0,
            maximum: 1,
            description: "Confidence score (0-1)"
          }
        },
        required: ["matches_tone", "confidence"]
      }
    });
    
    return [
      {
        dimension: "tone.match",
        value: json_payload.matches_tone ? 1.0 : 0.0,
        comment: `Target tone: ${targetTone}`
      } as Metric,
      {
        dimension: "tone.confidence",
        value: json_payload.confidence,
        comment: "Confidence in tone assessment"
      } as Metric
    ];
  }
);

Running Evaluations

Here's a simple example of running evaluators on an LLM response:

import asyncio
from opperai import AsyncOpper

opper = AsyncOpper()

async def main():
    # Get a response from the LLM
    answer, response = await opper.call(
        name="answer_question",
        instructions="Answer the question briefly and professionally",
        input="What is machine learning?"
    )
    
    print(f"LLM Response: {answer}")
    
    # Run evaluation
    evaluation = await opper.evaluate(
        span_id=response.span_id,
        evaluators=[
            word_count_evaluator(result=answer, min_words=5, max_words=50),
            tone_evaluator(result=answer, target_tone="professional")
        ]
    )
    
    # Print evaluation results
    print("\nEvaluation Results:")
    for group_name, metrics in evaluation.metrics.items():
        print(f"\n{group_name}:")
        for metric in metrics:
            print(f"  - {metric.dimension}: {metric.value:.2f} ({metric.comment})")

if __name__ == "__main__":
    asyncio.run(main())

import { Client } from "opperai";

const client = new Client();

async function main() {
  // Get a response from the LLM
  const { message, span_id } = await client.call({
    name: "answer_question",
    instructions: "Answer the question briefly and professionally",
    input: "What is machine learning?"
  });
  
  console.log(`LLM Response: ${message}`);
  
  // Run evaluation
  const evaluation = await client.evaluate({
    span_id,
    evaluators: [
      wordCountEvaluator(message, 5, 50),
      toneEvaluator(message, "professional")
    ]
  });
  
  // Print evaluation results
  console.log("\nEvaluation Results:");
  for (const [groupName, metrics] of Object.entries(evaluation.metrics)) {
    console.log(`\n${groupName}:`);
    for (const metric of metrics) {
      console.log(`  - ${metric.dimension}: ${metric.value.toFixed(2)} (${metric.comment})`);
    }
  }
}

main().catch(console.error);

Integration with Tracing

Evaluations are automatically associated with their corresponding trace spans, making it easy to see evaluation results in the context of your application's execution flow. This integration provides:

A complete picture of function performance across multiple calls
The ability to drill down into specific traces to understand evaluation results in context
Easy access to all metrics from a specific evaluation run

See Tracing for more information on how to use traces in your application.

Best Practices

When implementing evaluations in your application, consider these best practices:

Define clear success criteria: Determine what "good" looks like for your specific application and create evaluators that measure those aspects.
Combine multiple evaluators: Use multiple evaluators to measure different aspects of your function's performance (correctness, tone, format, etc.).
Start with simple metrics: Begin with simple, deterministic evaluators before moving to more complex LLM-based evaluations.
Use evaluations for regression testing: Run evaluations before and after making changes to ensure quality doesn't degrade.
Include examples from real user interactions: Add problematic outputs from real usage to your test dataset to ensure your function handles edge cases correctly.

More Examples

For more examples and patterns, check out:

Offline Evaluation

Offline evaluations let you systematically test your function against a dataset of input/output pairs (see Datasets).

Evaluation of a call using different models

Each dataset entry contains an input to test and the expected output to compare against. When you run an evaluation, Opper:

Executes your function on each input in the dataset
Compares the function's output to the expected output
Generates metrics on how well the outputs match expectations

Run an evaluation

To run an evaluation:

Navigate to a function's evaluation tab
Click "Run Evaluation" - this will be pre-filled with your current function configuration
Optionally modify the instructions, model, or other settings to test different combinations
Click run to start the evaluation against your dataset

The evaluation process typically takes between 30 seconds to a few minutes, depending on your dataset size.

Inspecting a run

Once complete, you'll see:

An overall score showing how well the outputs matched expectations across all entries
Individual results for each dataset entry showing:
- The model's actual output
- The expected output
- A detailed comparison highlighting any differences
- The evaluation score for that specific entry

This detailed breakdown helps you identify exactly where your function is performing well and where it may need improvements.

Online Evaluation

Online evaluation automatically assesses every function call in real-time. Using AI, it evaluates how well each output aligns with the function's instructions, input schema, and examples. This provides instant feedback on output quality and surfaces potential issues immediately.

When calls receive low evaluation scores, they make excellent candidates for your test dataset. Adding these edge cases with corrected outputs helps build comprehensive test coverage and guides improvements to your model selection and instructions.

The platform aggregates online evaluation scores to show your function's real-world performance. You can also view AI-generated suggestions for improving your instructions on the function overview page, which you can validate through offline evaluation.