evals

Creates, updates, deletes, gets or lists an evals resource.

Overview

Name	evals
Type	Resource
Id	openai.evals.evals

Fields

The following fields are returned by SELECT queries:

get

The evaluation

Name	Datatype	Description
id	string	Unique identifier for the evaluation.
name	string	The name of the evaluation. (example: Chatbot effectiveness Evaluation)
created_at	integer (unixtime)	The Unix timestamp (in seconds) for when the eval was created.
data_source_config	object	Configuration of data sources used in runs of the evaluation. (title: CustomDataSourceConfig)
metadata	object	Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. (x-oaiTypeLabel: map)
object	string	The object type. (eval) (default: eval)
testing_criteria	array	A list of testing criteria. (default: eval)

list

A list of evals

Name	Datatype	Description
id	string	Unique identifier for the evaluation.
name	string	The name of the evaluation. (example: Chatbot effectiveness Evaluation)
created_at	integer (unixtime)	The Unix timestamp (in seconds) for when the eval was created.
data_source_config	object	Configuration of data sources used in runs of the evaluation. (title: CustomDataSourceConfig)
metadata	object	Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters. (x-oaiTypeLabel: map)
object	string	The object type. (eval) (default: eval)
testing_criteria	array	A list of testing criteria. (default: eval)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get	select	eval_id	openai-organization, openai-project
list	select		after, limit, order, order_by, openai-organization, openai-project
create	insert	data_source_config, testing_criteria	openai-organization, openai-project
update	update	eval_id	openai-organization, openai-project
delete	delete	eval_id	openai-organization, openai-project

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
eval_id	string	The ID of the evaluation to delete.
after	string	Identifier for the last eval from the previous pagination request.
limit	integer	Number of evals to retrieve. Automatically applied from a SQL LIMIT clause - SELECT ... LIMIT 10 sends limit=10 on the wire. Setting it explicitly in a WHERE clause is not required.
openai-organization	string	Optionally scope the request to a specific organization (overrides the default associated with the API key). Addressable in SQL as openai_organization.
openai-project	string	Optionally scope the request to a specific project (overrides the default associated with the API key). Addressable in SQL as openai_project.
order	string	Sort order for evals by timestamp. Use asc for ascending order or desc for descending order.
order_by	string	Evals can be ordered by creation time or last updated time. Use created_at for creation time or updated_at for last updated time.

SELECT examples

get

The evaluation

SELECT
id,
name,
created_at,
data_source_config,
metadata,
object,
testing_criteria
FROM openai.evals.evals
WHERE eval_id = '{{ eval_id }}' -- required
AND "openai-organization" = '{{ openai-organization }}'
AND "openai-project" = '{{ openai-project }}'
;

list

A list of evals

SELECT
id,
name,
created_at,
data_source_config,
metadata,
object,
testing_criteria
FROM openai.evals.evals
WHERE after = '{{ after }}'
AND "order" = '{{ order }}'
AND order_by = '{{ order_by }}'
AND "openai-organization" = '{{ openai-organization }}'
AND "openai-project" = '{{ openai-project }}'
;

INSERT examples

create

No description available.

INSERT INTO openai.evals.evals (
name,
metadata,
data_source_config,
testing_criteria,
"openai-organization",
"openai-project"
)
SELECT 
'{{ name }}',
'{{ metadata }}',
'{{ data_source_config }}' /* required */,
'{{ testing_criteria }}' /* required */,
'{{ openai-organization }}',
'{{ openai-project }}'
RETURNING
id,
name,
created_at,
data_source_config,
metadata,
object,
testing_criteria
;

Manifest

# Description fields are for documentation purposes
- name: evals
props:
  - name: name
    value: "{{ name }}"
    description: |
      The name of the evaluation.
  - name: metadata
    value: "{{ metadata }}"
    description: |
      Set of 16 key-value pairs that can be attached to an object. This can be
      useful for storing additional information about the object in a structured
      format, and querying for objects via API or the dashboard.
      Keys are strings with a maximum length of 64 characters. Values are strings
      with a maximum length of 512 characters.
  - name: data_source_config
    description: |
      The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
    value:
      type: "{{ type }}"
      item_schema: "{{ item_schema }}"
      include_sample_schema: {{ include_sample_schema }}
      metadata: "{{ metadata }}"
  - name: testing_criteria
    value: "{{ testing_criteria }}"
    description: |
      A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like `{{item.variable_name}}`. To reference the model's output, use the `sample` namespace (ie, `{{sample.output_text}}`).
  - name: openai-organization
    value: "{{ openai-organization }}"
    description: Optionally scope the request to a specific organization (overrides the default associated with the API key). Addressable in SQL as `openai_organization`.
    description: Optionally scope the request to a specific organization (overrides the default associated with the API key). Addressable in SQL as `openai_organization`.
  - name: openai-project
    value: "{{ openai-project }}"
    description: Optionally scope the request to a specific project (overrides the default associated with the API key). Addressable in SQL as `openai_project`.
    description: Optionally scope the request to a specific project (overrides the default associated with the API key). Addressable in SQL as `openai_project`.

UPDATE examples

update

No description available.

UPDATE openai.evals.evals
SET 
name = '{{ name }}',
metadata = '{{ metadata }}'
WHERE 
eval_id = '{{ eval_id }}' --required
AND "openai-organization" = '{{ openai-organization}}'
AND "openai-project" = '{{ openai-project}}'
RETURNING
id,
name,
created_at,
data_source_config,
metadata,
object,
testing_criteria;

DELETE examples

delete

No description available.

DELETE FROM openai.evals.evals
WHERE eval_id = '{{ eval_id }}' --required
AND "openai-organization" = '{{ openai-organization }}'
AND "openai-project" = '{{ openai-project }}'
;