Indexes

What is an Index?

A Index is a set of documents that can be queried semantically in order to find similar documents. Our HTTP API supports low level indexing of text documents and file uploads. Opper will automatically chunk, generate embeddings and index the files, making them available for querying. Quering uses various strategies to retrieve the most relevant documents, such as semantic search with reranking. Filtering on meta data is also supported.

Create an Index

from opperai import AsyncOpper

opper = AsyncOpper()

async def main():
    index = await opper.indexes.get(name="my-index")
    if index is None:
        index = await opper.indexes.create(name="my-index")

    print(index.name)
    # my-index

asyncio.run(main())

import Client from "opperai";

const client = new Client();

(async () => {
    // Try to get the index first
    let index = await client.indexes.get("my-index");

    // If the index doesn't exist, create it
    if (!index) {
        index = await client.indexes.create("my-index");
    }

    console.log(index);
})();

Indexing data

Here we fetch an existing Index and index an object

from opperai import AsyncOpper
from opperai.types import DocumentIn

opper = AsyncOpper()

async def main():
    index = await opper.indexes.get(name="my-index")

    await index.add(
        key="document-name",
        content=f"This is some content that I want to index",
        metadata={
            "some_key": "some_value",
        },
    )

asyncio.run(main())

import Client from "opperai";

const client = new Client();

(async () => {
    // Get the index
    const index = await client.indexes.get("my-index");

    // Add documents to the index
    await index.add({
        key: "document-name",
        content: "This is some content that I want to index",
        metadata: { 
            "some_key": "some_value",
        }
    });
})();

Indexing files

It is also possible to upload files of the following types: PDF, CSV, Txt. And have Opper automatically chunk and index the files in an optimal way.

from opperai import AsyncOpper

opper = AsyncOpper()

async def main():
    index = await opper.indexes.get(name="my-index")
    await index.upload_file(file_path="tools/reddit-sec.pdf")

asyncio.run(main())

import Client from "opperai";

const client = new Client();

(async () => {
    // Get the index
    const index = await client.indexes.get("my-index");

    // Upload a file to the index
    const file = await index.uploadFile("examples/example.txt");

    console.log("file: ", file);
})();

For other file types or documents, the lower level index API can be used to upload and index the data.

Querying index

Querying an index is done by providing a query string and an optional filter. The query string is used to find the most semantically similar "chunks" in the index. A filter can optionally be used to narrow down the results.

from opperai import AsyncOpper

opper = AsyncOpper()

async def main():
    # Get the index
    index = await opper.indexes.get(name="my-index")

    # Query the index
    results = await index.query(
        query="What is the SEC filing for reddit?", 
        filter={"author": "John Doe"}, # optional filter
        k=5 # optional number of chunks to return
    )

    print(results)

asyncio.run(main())

import Client from "opperai";

const client = new Client();

(async () => {
    // Get the index
    const index = await client.indexes.get("my-index");

    // Query the index
    const results = await index.query({
        query: "What is the SEC filing for reddit?",
        // optional filter
        filter: {
            author: "John Doe"
        }, 
        // optional number of chunks to return
        k: 5, 
    });
})();

Using Retrieval Results

You can use the results of the query in a function call like this:

from opperai import AsyncOpper

opper = AsyncOpper()

async def main():
    index = await opper.indexes.get(name="my-index")

    # Query the index
    results = await index.query(
        query="What is the SEC filing for reddit?", 
        filter={"author": "John Doe"}, # optional filter
        k=5 # optional number of chunks to return
    )

    # Use the respond function with contents
    response, _ = await opper.call(
        name="answer_question",
        instructions="Answer the question using the knowledge",
        input={
            "question": "What revenue did Reddit report in 2023?",
            "knowledge": results
        },
        output_type=str
    )

    print(response)

asyncio.run(main())

import Client from "opperai";

const client = new Client();

(async () => {
    // Get the index
    const index = await client.indexes.get("my-index");

    // Query the index
    const results = await index.query({
        query: "What is the SEC filing for reddit?",
        // optional filter
        filter: {
            author: "John Doe"
        }, 
        // optional number of chunks to return
        k: 5, 
    });

    // Use the results in a function call
    const { message } = await client.call({
        name: "answer_question",
        instructions: "Answer the question using the knowledge",
        input: {
            question: "What revenue did Reddit report in 2023?",
            knowledge: results,
        },
    });

    console.log(message);
})();

Using the UI

On portal you can see all the indexes you have created. You can also use the portal to create, update and delete indexes.

Additional reading

Read our Cookbook for additional examples: https://github.com/opper-ai/opper-cookbook