Best Practices

If a muscle doesn't make music, don't use it.

-- Emanuel Feuermann

Building a good model is only one part of what's required to automate decisions. Those models must exist in a production environment and make the right decision under a variety of circumstances. This section provides some advice on how to get the most out of the models you've built.

Deployment pattern

Hop models are capable of being deployed to Docker containers, see deployment recipes for docker with CLI and HTTP runner configurations for docker.

Set timeouts

By default, Hop waits to finish its search before providing output. In production, a model needs to take action with the best solution it finds in a reasonable amount of time.

Hop supports terminating search early through the HOP_SOLVER_LIMITS_DURATION variable. Any model deployed into a production environment should have this configured. Set it using the Go syntax for specifying time durations. For example, a timeout of 90 seconds is 1m30s, while 100 milliseconds is 100ms.

Dash simulations run until there are no actors left in the simulation, or a timeout set by DASH_SIMULATOR_LIMITS_DURATION or DASH_SIMULATOR_TIME_DURATION is reached. As with HOP_SOLVER_LIMITS_DURATION, any simulation run in a production environment should have a timeout set to a value using GO's time duration syntax; i.e. 5h to run the simulation for five simulated hours or 5s to run for 5 real world seconds.

Consider request and response size

The default setting for HOP_RUNNER_OUTPUT_SOLUTIONS is last. This means that Hop will only output the final solution. If you want Hop to output every improving solutions it finds, set this variable to all. This setting could be helpful while you develop your model, for example. The number of solutions will vary by model settings, input data, and run duration.

We recommend setting the HOP_RUNNER_OUTPUT_STREAM variable to true while you develop your model. This variable displays solutions immediately after they are found, so you can better understand opportunities for improvement.

Similarly, we recommend turning both DASH_RUNNER_OUTPUT_MEASURES and DASH_RUNNER_OUTPUT_EVENTS on (that is, setting them to a value other than the default null) during development to aid in debugging your models. To make sure you can compare model output over time, it is also often helpful to write output data to a timestamped file:

echo input.json | ./queue > $(date '+%s').json # current unix time

In production, leaving DASH_RUNNER_OUTPUT_EVENTS unset is advised.

If you want to introduce randomness into your simulation to better understand model performance under variable conditions, you can specify a random seed integer value either via the -dash.simulator.random.seed CLI flag or the DASH_SIMULATOR_RANDOM_SEED environment variable. A convenient method for obtaining a random seed is to use -dash.simulator.random.seed $RANDOM in the shell.

Store inputs and outputs

Make sure to store the inputs and outputs used by your Hop models somewhere for analysis. This will let you rerun production executions later to answer questions, understand the emergent behavior of your automation, and study potential changes to your models.

If you are able to store the model output in a database, you can audit your model decisions and track the behavior of your automated decisions. Storing events and measures provided by a Dash simulation runner will enable visualization and analysis of simulation data as your models evolve.

Add model tags

The options passed to a solver or simulation can store arbitrary tags. Hop and Dash add these to the output JSON as a nested "tags" object:

{
  "tags": {
      "model_version": "v1.2.3",
      "speed_coefficient": 0.2
  },
}

Use this to add metadata and make your life easier when you need to dig into model or system behavior later.

func main() {
    cli.Run(
        func(in input, opt solve.Options) (solve.Solver, error) {
            opt.Tags["model_version"] = "v1.2.3"
            opt.Tags["speed_coefficient"] = 0.2
            return solve.Minimize(state(input), opt), nil
        },
    )
}

Good tags answer questions like:

• How did I estimate the distance or time between locations?
• Is there a marketing promotion impacting my operations?
• What is the version of my model?