runs

Creates, updates, deletes, gets or lists a runs resource.

Overview

Name	`runs`
Type	Resource
Id	`openai.evals.runs`

Fields

The following fields are returned by SELECT queries:

get
list

The evaluation run

Name	Datatype	Description
`id`	`string`	Unique identifier for the evaluation run.
`name`	`string`	The name of the evaluation run.
`eval_id`	`string`	The identifier of the associated evaluation.
`created_at`	`integer (unixtime)`	Unix timestamp (in seconds) when the evaluation run was created.
`data_source`	`object`	Information about the run's data source. (title: JsonlRunDataSource)
`error`	`object`	An object representing an error response from the Eval API. (title: EvalApiError)
`metadata`	`object`	Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. (x-oaiTypeLabel: map)
`model`	`string`	The model that is evaluated, if applicable.
`object`	`string`	The type of the object. Always "eval.run". (eval.run) (default: eval.run)
`per_model_usage`	`array`	Usage statistics for each model during the evaluation run.
`per_testing_criteria_results`	`array`	Results per testing criteria applied during the evaluation run.
`report_url`	`string (uri)`	The URL to the rendered evaluation run report on the UI dashboard.
`result_counts`	`object`	Counters summarizing the outcomes of the evaluation run.
`status`	`string`	The status of the evaluation run.

A list of runs for the evaluation

Name	Datatype	Description
`id`	`string`	Unique identifier for the evaluation run.
`name`	`string`	The name of the evaluation run.
`eval_id`	`string`	The identifier of the associated evaluation.
`created_at`	`integer (unixtime)`	Unix timestamp (in seconds) when the evaluation run was created.
`data_source`	`object`	Information about the run's data source. (title: JsonlRunDataSource)
`error`	`object`	An object representing an error response from the Eval API. (title: EvalApiError)
`metadata`	`object`	Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. (x-oaiTypeLabel: map)
`model`	`string`	The model that is evaluated, if applicable.
`object`	`string`	The type of the object. Always "eval.run". (eval.run) (default: eval.run)
`per_model_usage`	`array`	Usage statistics for each model during the evaluation run.
`per_testing_criteria_results`	`array`	Results per testing criteria applied during the evaluation run.
`report_url`	`string (uri)`	The URL to the rendered evaluation run report on the UI dashboard.
`result_counts`	`object`	Counters summarizing the outcomes of the evaluation run.
`status`	`string`	The status of the evaluation run.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params
`get`	`select`	`eval_id`, `run_id`	`openai-organization`, `openai-project`
`list`	`select`	`eval_id`	`after`, `limit`, `order`, `status`, `openai-organization`, `openai-project`
`create`	`insert`	`eval_id`, `data_source`	`openai-organization`, `openai-project`
`delete`	`delete`	`eval_id`, `run_id`	`openai-organization`, `openai-project`
`cancel`	`exec`	`eval_id`, `run_id`	`openai-organization`, `openai-project`

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
`eval_id`	`string`	The ID of the evaluation whose run you want to cancel.
`run_id`	`string`	The ID of the run to cancel.
`after`	`string`	Identifier for the last run from the previous pagination request.
`limit`	`integer`	Number of runs to retrieve. Automatically applied from a SQL `LIMIT` clause - `SELECT ... LIMIT 10` sends `limit=10` on the wire. Setting it explicitly in a `WHERE` clause is not required.
`openai-organization`	`string`	Optionally scope the request to a specific organization (overrides the default associated with the API key). Addressable in SQL as `openai_organization`.
`openai-project`	`string`	Optionally scope the request to a specific project (overrides the default associated with the API key). Addressable in SQL as `openai_project`.
`order`	`string`	Sort order for runs by timestamp. Use `asc` for ascending order or `desc` for descending order. Defaults to `asc`.
`status`	`string`	Filter runs by status. One of `queued` \| `in_progress` \| `failed` \| `completed` \| `canceled`.

`SELECT` examples

get
list

The evaluation run

SELECT
id,
name,
eval_id,
created_at,
data_source,
error,
metadata,
model,
object,
per_model_usage,
per_testing_criteria_results,
report_url,
result_counts,
status
FROM openai.evals.runs
WHERE eval_id = '{{ eval_id }}' -- required
AND run_id = '{{ run_id }}' -- required
AND "openai-organization" = '{{ openai-organization }}'
AND "openai-project" = '{{ openai-project }}'
;

A list of runs for the evaluation

SELECT
id,
name,
eval_id,
created_at,
data_source,
error,
metadata,
model,
object,
per_model_usage,
per_testing_criteria_results,
report_url,
result_counts,
status
FROM openai.evals.runs
WHERE eval_id = '{{ eval_id }}' -- required
AND after = '{{ after }}'
AND "order" = '{{ order }}'
AND status = '{{ status }}'
AND "openai-organization" = '{{ openai-organization }}'
AND "openai-project" = '{{ openai-project }}'
;

`INSERT` examples

create
Manifest

No description available.

INSERT INTO openai.evals.runs (
name,
metadata,
data_source,
eval_id,
"openai-organization",
"openai-project"
)
SELECT 
'{{ name }}',
'{{ metadata }}',
'{{ data_source }}' /* required */,
'{{ eval_id }}',
'{{ openai-organization }}',
'{{ openai-project }}'
RETURNING
id,
name,
eval_id,
created_at,
data_source,
error,
metadata,
model,
object,
per_model_usage,
per_testing_criteria_results,
report_url,
result_counts,
status
;

# Description fields are for documentation purposes
- name: runs
props:
  - name: eval_id
    value: "{{ eval_id }}"
    description: Required parameter for the runs resource.
  - name: name
    value: "{{ name }}"
    description: |
      The name of the run.
  - name: metadata
    value: "{{ metadata }}"
    description: |
      Set of 16 key-value pairs that can be attached to an object. This can be
      useful for storing additional information about the object in a structured
      format, and querying for objects via API or the dashboard.
      Keys are strings with a maximum length of 64 characters. Values are strings
      with a maximum length of 512 characters.
  - name: data_source
    description: |
      Details about the run's data source.
    value:
      type: "{{ type }}"
      source:
        type: "{{ type }}"
        content:
          - item: "{{ item }}"
            sample: "{{ sample }}"
        id: "{{ id }}"
      input_messages:
        type: "{{ type }}"
        template: "{{ template }}"
        item_reference: "{{ item_reference }}"
      sampling_params:
        reasoning_effort: "{{ reasoning_effort }}"
        temperature: {{ temperature }}
        max_completion_tokens: {{ max_completion_tokens }}
        top_p: {{ top_p }}
        seed: {{ seed }}
        response_format: "{{ response_format }}"
        tools:
          - type: "{{ type }}"
            function:
              description: "{{ description }}"
              name: "{{ name }}"
              parameters: "{{ parameters }}"
              strict: {{ strict }}
      model: "{{ model }}"
  - name: openai-organization
    value: "{{ openai-organization }}"
    description: Optionally scope the request to a specific organization (overrides the default associated with the API key). Addressable in SQL as `openai_organization`.
    description: Optionally scope the request to a specific organization (overrides the default associated with the API key). Addressable in SQL as `openai_organization`.
  - name: openai-project
    value: "{{ openai-project }}"
    description: Optionally scope the request to a specific project (overrides the default associated with the API key). Addressable in SQL as `openai_project`.
    description: Optionally scope the request to a specific project (overrides the default associated with the API key). Addressable in SQL as `openai_project`.

`DELETE` examples

delete

No description available.

DELETE FROM openai.evals.runs
WHERE eval_id = '{{ eval_id }}' --required
AND run_id = '{{ run_id }}' --required
AND "openai-organization" = '{{ openai-organization }}'
AND "openai-project" = '{{ openai-project }}'
;

Lifecycle Methods

cancel

The canceled eval run object

EXEC openai.evals.runs.cancel 
@eval_id='{{ eval_id }}' --required, 
@run_id='{{ run_id }}' --required, 
@openai-organization='{{ openai-organization }}', 
@openai-project='{{ openai-project }}'
;

Overview​

Fields​

Methods​

Parameters​

SELECT examples​

INSERT examples​

DELETE examples​

Lifecycle Methods​