To try out shadow experiments with Nextmv, you must have a Nextmv account and be signed up for a free trial or on an active plan. For questions, please contact Nextmv support or start a free trial.
This page assumes you have read the Experiments Core Concept page.
A shadow test is an experiment that runs in the background and compares the results of a baseline instance against a candidate instance. When the shadow test has started, any run made on the baseline instance will trigger a run on the candidate instance using the same input and options. The results of the shadow test are often used to determine if a new version of a model is ready to be promoted to production.
Creating a shadow test
Shadow tests can be created from Nextmv CLI or Nextmv Console. The required entities to start a shadow test are described in the table below.
| Field | Description |
|---|---|
| Name & description | These are for your reference only and have no bearing on the test. The name is required, the description is optional. |
| ID | The ID may only include lowercase letters, numbers, periods, or hyphens and must be between 3 and 30 characters (and cannot start or end with a hyphen or period). The ID must be unique to the app the test is located in as well. |
| Baseline instance | The instance to use for the baseline instance. When the shadow test has started, any run made on the baseline instance will trigger a run on the candidate instance using the same input. Note that a baseline instance can only have one active shadow test running at any given time. |
Start time (start criteria) | If desired, you can specify a date and time for the test to start collecting data. If defined, the test will begin collecting data on this date when the test is in a started state. The test can be started manually before the start time, but it will not begin collecting data until this start date. However, note that a test WILL NOT START AUTOMATICALLY if a start time is defined. You must still manually start the test. |
| End time (end criteria) | An optional end date and time can be given for the shadow test. The test will end on this date and time unless the maximum number of runs have been met first. (You can also manually complete or cancel a shadow test at any time). |
| Maximum runs (end criteria) | A maximum number of baseline runs can be set for the shadow test. The test will end once this maximum number of runs has been reached (unless an end date and time was set and this was reached first). There is a maximum run limit of 50 baseline runs (100 total runs: 50 baseline, 50 candidate) no matter which end criteria is set. |
Using Nextmv CLI
You can create a batch experiment using the Nextmv CLI:
Executing the create command will create the test in a draft state (the return from the CLI command will be the test ID). The details of the test in its draft state can be viewed in Console or with Nextmv CLI using the result command.
To start the shadow test with Nextmv CLI you must run the nextmv experiment shadow start command. After the test has been started, the results of the test can be viewed with Nextmv CLI or in Nextmv Console. The test will display partial results as the data is processed so you do not have to wait until the test is complete to review the results.
For a full reference on using Nextmv CLI commands for managing shadow tests visit the CLI reference page.
Using Nextmv Console
Navigate to the Experiments section in your app, click on the Shadow page in the expanded subsection, and then click on the New Shadow Test button. Fill in the fields and then you can either click Create & start shadow test or Create draft shadow test. The former creates the shadow test and then immediately sets the status to started. The latter creates the shadow test but leaves it in a draft state.
You will be returned to the list of shadow tests where you can then click on the newly create test to view the details. From the details page you can manually start the test if it was created in draft state.
Viewing a shadow test
The results of a shadow test can be retrieved via Nextmv CLI or viewed in Nextmv Console. Nextmv CLI returns the raw data in JSON format and is useful if you would like to perform your own operations on the test results. Nextmv Console displays the results of the shadow test in a web interface on the test details view.
The results of the shadow test are returned in the same structure as a batch experiment. And the runs from a shadow test are also accessed in the same way they are for a batch experiment as well. See the Structure of result and Experiment runs sections in the batch experiment reference to learn more about analyzing the results of a shadow test.