Updated acceptance tests

When creating a new acceptance test you can now specify a tolerance level for the pass/fail rules. Prior to this, you were only able to set hard rules on how a test would pass or fail. When setting the tolerance, you can define it as an absolute value (specific number) or as a relative value (percentage-based).

Screenshot of the UI in the create new acceptance test view that allows you to type in the metric you want to track and then define the tolerance type (absolute or relative) and the corresponding value.

This tolerance will be reflected in the final chart visuals as well.

Screenshot of a point and line chart that plots a target line running diagonally in the middle from lower left to upper right with a band of green on either side that visualizes the threshold for acceptable values within range of the target line. Points that correspond to the runs are plotted on top of this band to show which runs fall inside and outside of the threshold.

Updated acceptance tests

Define tolerance thresholds for pass/fail and more