Launch an `Evaluation` job which evaluates an `Agent` on a set of test questions and reference answers.

An `Evaluation` is an asynchronous operation. Users can select one or more metrics to assess the quality of generated answers. These metrics include `equivalence` and `groundedness`. `equivalence` evaluates if the Agent response is equivalent to the ground truth (model-driven binary classification). `groundedness` decomposes the Agent response into claims and then evaluates if the claims are grounded by the retrieved documents.

`Evaluation` data can be provided in one of two forms:

- A CSV `evalset_file` containing the columns `prompt` (i.e. questions) and `reference` (i.e. gold-answers).

- An `evalset_name` which refers to a `Dataset` created through the `/datasets/evaluate` API.

Create Evaluation

EvaluationMetric

Location

Message

Error Type

ValidationError

Body_create_evaluation_agents__agent_id__evaluate_post

BearerAuth

LaunchEvaluationResponse

HTTPValidationError

Contextual AI Documentation

Welcome to the Contextual AI Documentation

Beginner's Guide

Learn how to tune and evaluate your agent

Tuning and Evaluation

This guide explains how to install and use the Contextual AI Snowflake Native App

Snowflake Native Application

Billing and Subscriptions management guide for self-serve customers

Billing and Subscriptions

Welcome to the home of your new documentation

Key Concepts

Python SDK

Node.js SDK

Retrieve a list of `Datastores`.

Performs `cursor`-based pagination if the number of `Datastores` exceeds the requested `limit`. The returned `cursor` can be passed to the next `GET /datastores` call to retrieve the next set of `Datastores`.

List Datastores

Create a new `Datastore`.

A `Datastore` is a collection of documents. Documents can be ingested into and deleted from a `Datastore`.

A `Datastore` can be linked to one or more `Agents`, and conversely, an `Agent` can be associated with one or more `Datastores` to ground its responses with relevant data. This flexible many-to-many relationship allows `Agents` to draw from multiple sources of information. This linkage of `Datastore` to `Agent` is done through the `Create Agent` or `Edit Agent` APIs.

> Note that self-serve users are currently required to create datastores through our UI. Otherwise, they will receive the following message: "This endpoint is disabled as you need to go through checkout. Please use the UI to make this request."

Create Datastore

Get the details of a given `Datastore`, including its name, create time, and the list of `Agents` which are currently configured to use the `Datastore`.

Get Datastore Metadata

Delete a given `Datastore`, including all the documents ingested into it. This operation is irreversible.

This operation will fail with status code 400 if there is an active `Agent` associated with the `Datastore`.

Delete Datastore

Get list of documents in a given `Datastore`, including document `id`, `name`, and ingestion job `status`.

Performs `cursor`-based pagination if the number of documents exceeds the requested `limit`. The returned `cursor` can be passed to the next `GET /datastores/{datastore_id}/documents` call to retrieve the next set of documents.

List Documents

Ingest a document into a given `Datastore`.

Ingestion is an asynchronous task. Returns a document `id` which can be used to track the status of the ingestion job through calls to the `GET /datastores/{datastore_id}/documents/{document_id}/metadata` API.

This `id` can also be used to delete the document through the `DELETE /datastores/{datastore_id}/documents/{document_id}` API.

`file` must be a PDF, HTML, DOC(X) or PPT(X) file. The filename must end with one of the following extensions: `.pdf`, `.html`, `.htm`, `.mhtml`, `.doc`, `.docx`, `.ppt`, `.pptx`.

Ingest Document

Get details of a given document, including its `name` and ingestion job `status`.

Get Document Metadata

Post details of a given document that will enrich the chunk and be added to the context or just for filtering. If Just for filtering, start with "_" in the key.

Update Document Metadata

Delete a given document from its `Datastore`. This operation is irreversible.

Delete Document

List Agents

Create a new `Agent` with a specific configuration.

This creates a specialized RAG `Agent` which queries over one or multiple `Datastores` to retrieve relevant data on which its generations are grounded.

Retrieval and generation parameters are defined in the provided `Agent` configuration.

If no `datastore_id` is provided in the configuration, this API automatically creates an empty `Datastore` and configures the `Agent` to use the newly created `Datastore`.

> Note that self-serve users are currently required to create agents through our UI. Otherwise, they will receive the following message: "This endpoint is disabled as you need to go through checkout. Please use the UI to make this request."

Create Agent

Modify a given `Agent` to utilize the provided configuration.

Fields not included in the request body will not be modified.

Edit Agent

Delete a given `Agent`. This is an irreversible operation.

Note: `Datastores` which are associated with the `Agent` will not be deleted, even if no other `Agent` is using them. To delete a `Datastore`, use the `DELETE /datastores/{datastore_id}` API.

Delete Agent

Get metadata and configuration of a given `Agent`.

Get Agent Metadata

Start a conversation with an `Agent` and receive its generated response, along with relevant retrieved data and attributions.

Query

Return metadata of the contents used to generate the response for a given message.

Get Retrieval Info

Provide feedback for a generation or a retrieval. Feedback can be used to track overall `Agent` performance through the `Feedback` page in the Contextual UI, and as a basis for model fine-tuning.

If providing feedback on a retrieval, include the `message_id` from the `/query` response, and a `content_id` returned in the query's `retrieval_contents` list.

For feedback on generations, include `message_id` and do not include a `content_id`.

Provide Feedback

Returns usage and user-provided feedback data. This information can be used for data-driven improvements and optimization.

Get Metrics

Retrieve a list of `Evaluation` jobs run for a given `Agent`, including the `Evaluation`'s status and other metadata.

List Evaluations

Get an `Evaluation` job's status and results. There are six possible statuses: 'pending', 'processing', 'retrying', 'completed', 'failed', 'cancelled'.

If the evaluation job has completed, you will see your evaluation `metrics` , `job_metadata`, and the `dataset_name` where your eval metrics and row-by-row results are stored. You can use the `/datasets/evaluate` API to view the specified `dataset`.

Get Evaluation Metadata

Cancels an `Evaluation` job if it is still in progress.

Cancel Evaluation

List all evaluation `Datasets` and their versions belonging to a particular `Agent`.

If a `dataset_name` filter is provided, all versions of that `Dataset` will be listed.

Includes metadata and schema for each `Dataset` version.

List Evaluation Datasets

Create a new evaluation `Dataset` for the specified `Agent` using the provided JSONL or CSV file. A `Dataset` is a versioned collection of samples conforming to a particular schema, and can be used to store `Evaluation` test-sets and retrieve `Evaluation` results.

Each `Dataset` is versioned and validated against its schema during creation and subsequent updates. The provided `Dataset` file must conform to the schema defined for the `dataset_type`.

File schema for `dataset_type` `evaluation_set` is a CSV file or a JSONL file where each line is one JSON object. The following keys are required:

- `prompt` (`string`): Prompt or question

- `reference` (`string`): Reference or ground truth response

Create Evaluation Dataset

Stream the raw content of an evaluation `Dataset` version. If no version is specified, the latest version is used.

The `Dataset` content is downloaded in batches. Batch size can be configured to meet specific processing requirements.

Returns a `StreamingResponse`, an asynchronous stream of `Dataset` content with:

 - Content-Type: application/octet-stream

 - Content-Disposition: attachment

 - Chunked transfer encoding

Get Evaluation Dataset

Append to an existing evaluation `Dataset`.

Create a new version of the dataset by appending content to the `Dataset` and validating against its schema.

File schema for `dataset_type` `evaluation_set` is a CSV file or a JSONL file where each line is one JSON object. The following keys are required:

- `prompt` (`string`): Prompt or question

- `reference` (`string`): Reference or ground truth response

Append To Evaluation Dataset

Delete an evaluation `Dataset` and all its versions.

Permanently removes the `Dataset`, including all associated metadata.

This operation is irreversible.

Delete Evaluation Dataset

Retrieve details of a specific evaluation `Dataset` version, or the latest version if no `version` is specified.

Provides comprehensive information about the `Dataset`, including its metadata and schema.

Get Evaluation Dataset Metadata

List all tuning `Datasets` and their versions belonging to a particular `Agent`.

If a `dataset_name` filter is provided, all versions of that `Dataset` will be listed.

Includes metadata and schema for each `Dataset` version.

List Tuning Datasets

Create a new tuning `Dataset` for the specified `Agent` using the provided JSONL or CSV file. A `Dataset` is a versioned collection of samples conforming to a particular schema, and can be used as a source of training and test data for tuning jobs.

Each `Dataset` is versioned and validated against its schema during creation and subsequent updates. The provided `Dataset` file must conform to the schema defined for the `dataset_type`.

File schema for `dataset_type` `tuning_set` is a CSV file or a JSONL file where each line is one JSON object. The following keys are required:

- `knowledge` (`list[str]`):  Retrieved knowledge used to generate the reference answer. `knowledge` is a list of retrieved text chunks.


- `reference` (`str`): The gold-standard answer to the prompt.

- `guideline` (`str`): Guidelines for model output.  If you do not have special guidelines for the model's output, you can use the `System Prompt` defined in your Agent configuration as the `guideline`.

- `prompt` (`str`): Question for the model to respond to.

For examples of what `tuning_set` should look like, check out our `Tune & Evaluation Guide`.

Create Tuning Dataset

Stream the raw content of a tuning `Dataset` version. If no version is specified, the latest version is used.

The `Dataset` content is downloaded in batches. Batch size can be configured to meet specific processing requirements.

Returns a `StreamingResponse`, an asynchronous stream of `Dataset` content with:

 - Content-Type: application/octet-stream

 - Content-Disposition: attachment

 - Chunked transfer encoding

Get Tuning Dataset

Append to an existing tuning `Dataset`.

Create a new version of the dataset by appending content to the `Dataset` and validating against its schema.

File schema for `dataset_type` `evaluation_set` is a CSV file or a JSONL file where each line is one JSON object. The following keys are required:

- `knowledge` (`list[str]`):  Retrieved knowledge used to generate the reference answer. `knowledge` is a list of retrieved text chunks.


- `reference` (`str`): The gold-standard answer to the prompt.

- `guideline` (`str`): Guidelines for model output.  If you do not have special guidelines for the model's output, you can use the `System Prompt` defined in your Agent configuration as the `guideline`.

- `prompt` (`str`): Question for the model to respond to.

For examples of what `tuning_set` should look like, check out our `Tune & Evaluation Guide`.

Append To Tuning Dataset

Delete a tuning `Dataset` and all its versions.

Permanently removes the `Dataset`, including all associated metadata.

This operation is irreversible.

Delete Tune Dataset

Retrieve details of a specific tuning `Dataset` version, or the latest version if no `version` is specified.

Provides comprehensive information about the `Dataset`, including its metadata and schema.

Get Tuning Dataset Metadata

Create a tuning job for the specified `Agent` to specialize it to your specific domain or use case.

This API initiates an asynchronous tuning task. You can provide the required data through one of two ways:

- Provide a `training_file` and an optional `test_file`. If no `test_file` is provided, a portion of the `training_file` will be held out as the test set. For easy reusability, the `training_file` is automatically saved as a `Tuning` `Dataset`, and the `test_file` as an `Evaluation` `Dataset`. You can manage them via the `/datasets/tune` and `/datasets/evaluation` endpoints.

- Provide a `Tuning` `Dataset` and an optional `Evaluation` `Dataset`. You can create a `Tuning` `Dataset` and `Evaluation` `Dataset` using the `/datasets/tune` and `/datasets/evaluation` endpoints respectively.

The API returns a tune job `id` which can be used to check on the status of your tuning task through the `GET /tune/jobs/{job_id}/metadata` endpoint.

After the tuning job is complete, the metadata associated with the tune job will include evaluation results and a model ID. You can then deploy the tuned model to the agent by editing its config with the tuned model ID and the "Edit Agent" API (i.e. the `PUT /agents/{agent_id}` API). To deactivate the tuned model, you will need to edit the Agent's config again and set the `llm_model_id` field to "default". For an end-to-end walkthrough, see the `Tune & Evaluation Guide`.

Submit Training Job

Retrieve a list of all fine-tuning jobs for a specified Agent.

List Tune Jobs

Retrieve the status of a specific tuning job. Fetches the current status and evaluation results, if available, for the specified tuning job. After the tuning job is complete, the metadata associated with the tune job will include evaluation results and a model ID. You can then activate the tuned model for your agent by editing its config with the tuned model ID and the "Edit Agent" API (i.e. the `PUT /agents/{agent_id}` API). To deactivate the tuned model, you will need to edit the Agent's config again and set the `llm_model_id` field to "default". For an end-to-end walkthrough, see the `Tune & Evaluation Guide`.

Get Tune Job

Cancel a specific fine-tuning job. Terminates the fine-tuning job if it is still in progress.

Cancel Tune Job

Retrieves a list of tuned models associated with the specified Agent.

List Tuned Models

Given a `query`, `response`, and a `unit_test`, return the response's `score` on the unit test on a 5-point continuous scale. The total input cannot exceed 7000 tokens.

See a code example in [our blog post](https://contextual.ai/news/lmunit/). Email [lmunit-feedback@contextual.ai](mailto:lmunit-feedback@contextual.ai) with any feedback or questions.

>🚀 Obtain an LMUnit API key by completing [this form](https://contextual.ai/request-lmunit-api/)

LMUnit

Get Users

Modify a given `User`.

Fields not included in the request body will not be modified.

Update User

Invite users to the tenant. This checks if the user is already in the tenant and if not, creates the user. We will return a list of user emails that were successfully created (including existing users).

Invite Users

Remove User

Generate a response using Contextual's Grounded Language Model (GLM), an LLM engineered specifically to prioritize faithfulness to in-context retrievals over parametric knowledge to reduce hallucinations in Retrieval-Augmented Generation and agentic use cases.

The total request cannot exceed 32,000 tokens.

See our [blog post](https://contextual.ai/blog/introducing-grounded-language-model/) and [code examples](https://colab.research.google.com/github/ContextualAI/examples/blob/main/03-standalone-api/02-generate/generate.ipynb). Email [glm-feedback@contextual.ai](mailto:glm-feedback@contextual.ai) with any feedback or questions.

Generate

Rank a list of documents according to their relevance to a query primarily and your custom instructions secondarily.  We evaluated the model on instructions for recency, document type, source, and metadata, and it can generalize to other instructions as well.

The total request cannot exceed 400,000 tokens. The combined length of the query, instruction and any document with its metadata must not exceed 8,000 tokens.

See our [blog post](https://contextual.ai/blog/introducing-instruction-following-reranker/) and [code examples](https://colab.research.google.com/github/ContextualAI/examples/blob/main/03-standalone-api/03-rerank/rerank.ipynb). Email [rerank-feedback@contextual.ai](mailto:rerank-feedback@contextual.ai) with any feedback or questions.

/datastores

/datastores/{id}/documents

/agents

/agents/{id}/query

/agents/{id}/evaluate

/agents/{id}/datasets/evaluate

/agents/{id}/datasets/tune

/agents/{id}/tune

/lmunit

/users

/generate

/rerank

Create Evaluation

Authorizations

Path Parameters

Body

Response