Skip to main content
The tests.json file allows you to specify your Openlayer tests. This allows you to fully control your project’s tests with a single configuration file that can be version-controlled, without needing to use the Openlayer UI to create and update tests. This guide shows how you can write the tests.json for your project.

​
Discoverability

First, it is important to signal to Openlayer that you have a test configuration file. You can do so in the testsPath field of your openlayer.json. Make sure that you point to your tests.json file, as in:
openlayer.json

​
Structure

The tests.json file should contain an array of Test objects, each representing a test in your project. The Test objects have a set of common attributes and a set of attributes that depend on the mode (i.e., if it is a development test or monitoring test).
The best way to write a test configuration is to copy the examples from the documentation and edit them to fit your use case. Each test has a documentation page with information about it and configuration examples. They are all listed here.

​
Test object common attributes

The common attributes must always be present, regardless of the mode.

​
name

Type: string, required The test name.

​
description

Type: string. The test description.

​
type

Type: string, required. The test type, which represents the test category. Must be one of integrity, consistency, or performance.

​
subtype

Type: string, required. The test subtype, which identifies the test on the platform. Must be one of the valid subtypes.

​
thresholds

Type: array of Threshold objects, required. The thresholds that

​
insightName

Type: string, required.Name of the insight from which the test is based. Must be one of the valid insight names.

​
measurement

Type: string, required.Key from the insightName on top of which the threshold will be applied. Must be one of the valid measurement names.

​
operator

Type: string, required.Operator used to compare the measurement and value. Must be one of is, >, >=, <, <=.

​
value

Type: number | string, required.Threshold value.

​
insightParameters

Type: object.Parameters needed to compute the insight. Might be null depending on the insight.

​
subpopulationFilters

Type: object. Filters that define the subpopulation.

​
tags

Type: array of string. The names of the tags the test belongs to — the same tags you can add to a test in the platform. Any name your project doesn’t have yet is created on push, so you don’t have to create the tags in the UI first:
tests.json
Tags are matched by exact name, so Regression and regression are two different tags. How a push treats a test’s tags depends on whether the field is present at all: Because the field is only read when it’s present, you can tag some tests from tests.json and leave others to be tagged in the UI.
openlayer tests writes the tags field for every test it exports, so you can start from the tests you already have on the platform and edit the tag names from there.

​
syncId

Type: string, required. An id (UUID) that identifies the test. It is stable across commits — Openlayer uses it to decide whether a push updates an existing test or creates a new one — so it is also the handle you use to remove tests with openlayer tests delete --sync-id.

​
mode

Type: string, required. Defines to which mode the test applies to. Must be one of development or monitoring.

​
Test object mode-specific attributes

The additional attributes you must specify for a test depend on its mode, i.e., if it is a development or monitoring mode test.

​
usesValidationDataset

Type: bool, requiredIndicates if the test uses the validation dataset.

​
usesTrainingDataset

Type: bool, requiredIndicates if the test uses the training dataset.

​
usesMlModel

Type: bool, requiredIndicates if the test uses model.