Tests and evals

Since LLMs are fundamentally probabilistic we typically need to spend some degree of effort to perform quality assurance on the completions. Opper offers a variety of ways of doing tests, from saving metrics to testing models.

​

How it works

Tests and evaluations are essentially about measuring the quality of the task completion. Since tasks are declarative and very structured they are much more easily evaluated than conventional LLM completions based on only prompts. In addition to enforcing schemas, promoting the defining of requirements in schemas, Opper also performs automatic observations using AI on every completion. You can also attach custom metrics to completions or spans.

​

Built in observations

The platform automatically performs an observation on the quality of the task completion. The results of these are available in the tracing view next to every task completion. The observation appears within 1-10 seconds. You will see a paragraph summary observation and a score from 0-100.

The built in observation is essentially a task in itself that completes into:

class Score(BaseModel):
    thoughts: str = Field(
        description="Thoughts on how to evaluate the response",
    )
    observations: str = Field(
        description="Observations about the operation and the response",
    )
    correct: bool = Field(
        description="Did the model succeed at handling the task or not?",
    )
    score: float = Field(
        description="A value between 0 and 100 reflecting the quality of the response given the instructions, input and expected output",
    )

​

Custom metrics

You may also attach custom metrics to task completions or spans.

Here is a very simple example:

from opperai import Opper
from pydantic import BaseModel, Field
import os

def main():
    opper = Opper(http_bearer=os.getenv("OPPER_API_KEY"))

    result = opper.call(
        name="answer-question",
        instructions="Answer the question as concisely as possible",
        input="What is the capital of France?"
    )

    # Perform a simple evaluation
    is_one_word = 1 if len(result.message.split(" ")) == 1 else 0

    if is_one_word:
        print("The answer is concise")
  
        opper.span_metrics.create_metric(
            span_id=result.span_id,
            dimension="is_concise", 
            value=1, 
            comment="Evaluated if the answer is concise (1=True, 0=False)"
        )
    else:
        print("The answer is not concise")
  
        opper.span_metrics.create_metric(
            span_id=result.span_id,
            dimension="is_concise", 
            value=0, 
            comment="Evaluated if the answer is concise (1=True, 0=False)"
        )

if __name__ == "__main__":
    main()

​

Run dataset evaluation

To test new models, prompts or other configuration it is possible to utilize the task datasets to run evaluations.

To run through a task with an alternative configuration, go into the dashboard, navigate to the function you want to test, go to evaluations and press “run”. You will be presented with options for changing the current configuration, including model and prompt.