Batch Experiments

Batch experiments are used to analyze the output from one or more decision models on a fixed set of inputs. They are generally used as an exploratory test to understand the impacts to business metrics (or KPIs) when updating a model with a new feature, such as an additional constraint. They can also be used to validate that a model is ready for further testing and likely to make an intended business impact.

Creating a batch experiment

Batch experiments can be created from Nextmv CLI or Nextmv Console. You can create a batch experiment without specifying an input set or instances, but it is best to be explicit when defining which models to test and which input set to use. The required entities to start a batch experiment are described in the table below.

Using Nextmv CLI

You can create a batch experiment using the Nextmv CLI:

nextmv experiment batch start \
  --app-id "pizza-delivery" \
  --experiment-id "pizza-delivery-region-compare" \
  --name "Pizza Delivery Region Compare" \
  --description "Compares NYC and Philly region models" \
  --input-set-id "sample-day-runs" \
  --instance-ids "nyc-pizza-region,philly-pizza-region"

After executing the command you will be prompted with a summary of how many runs will be performed for the experiment and if you want to continue. Entering y will return the experiment ID and start the batch experiment. The results of the batch experiment can be viewed with Nextmv CLI or in Nextmv Console.

For a full reference on using Nextmv CLI commands for managing batch experiments visit the CLI reference page.

Using Nextmv Console

Navigate to the Experiments section in your app (it will land on Batch as the default view) and then click on the New Experiment button. Fill in the fields and then click Create & run experiment.

You will be returned to the list of experiments where you can then click on the newly created experiment to view the results. Note, when running large experiments, you may need to check back later to view the results.

Viewing a batch experiment

The results of a batch experiment can be retrieved via Nextmv CLI or viewed in Nextmv Console. Nextmv CLI returns the raw data in JSON format and is useful if you would like to perform your own operations on the experiment results.

Nextmv Console displays the results grouped by indicator keys and for each one includes a summary table, the percentile values and a box plot chart displaying summary values from the versions compared. To view the batch experiment result in Console, navigate to your app and then click on the Experiments section of your app. You can type the name of your batch experiment in the filter box at the top of the experiment list or scroll to find it. Click on the experiment name and the experiment details view will load.

Structure of result

The result of a batch experiment is returned as JSON. If you’re viewing the result in Console, the JSON is parsed and displayed in a web view; Nextmv CLI returns the complete JSON response. The response contains the batch experiment metadata and the results in the form of grouped distributional summaries.

Top-level properties

The table below summarizes the top-level properties in the return.

Grouped distributional summaries

There are three types of summaries included in the grouped_distributional_summaries array (there could be more in the future):

1. Version (instance)
2. Version (instance) + input
3. Input

No matter the type, each grouped distributional summary includes the following:

Indicator distributions

Each object property value in the indicator_distributions contains the values in the table below. Note that for some runs, certain values may be missing (a custom statistic for example). If you’re viewing the results in Console and a grouped distributional summary is missing values, a warning message will appear. If you are analyzing the results from the returned JSON, you must handle this check in your own systems.

Experiment runs

Runs made for the experiment can be retreived with the /applications/{application_id}/experiments/batch/{batch_id}/runs endpoint. This will return the runs of the experiment in the following format:

{
  "runs": [
    {
        "id": "staging-dehnyogSg",
        "input_id": "staging-wZ8xcIQ4R",
        "created_at": "2023-08-21T21:09:41.48243386Z",
        "application_id": "pizza-delivery",
        "experiment_id": "pizza-delivery-region-compare",
        "application_instance_id": "nyc-pizza-region",
        "application_version_id": "nyc-v1.2.13",
        "status": "succeeded",
        "statistics": {
          "status": "succeeded",
          "indicators": [
            {
              "name": "result.value",
              "value": 2175064,
            },
            ...
          ]
        }
    },
    ...
  ]
}

Where runs is an array that contains the runs made for the experiment. Each run object includes the run metadata plus any summary statistics that were specified in the excecutable binaries used for the experiment (see table below).

Experiment runs must be retrieved independent of the batch experiment details. However, in Console this run history table can be viewed at the bottom of the experiment details. Each run history item can be clicked to view the details of the run. If the app is a routing app (using either the routing or nextroute template) the run details will also include a visualization of the results.

The run history data can also can be downloaded as a CSV file in Console. Click the Download CSV link in the upper right area of the experiment run history table to download the data as a CSV file.

For more information on retrieving experiment runs, visit the Experiments API reference.

Experiment run summary data

Run history objects can include the items specified in the table below.