What are the considerations for when to use a switchback test vs other types of decision model testing?
You should use a switchback test to account for network effects when you want to see how a candidate model makes decisions that impact real-world operations. It’s best practice to run a series of historical tests such as batch, scenario, and acceptance tests, and then progress to shadow testing on production data. Switchback testing is often a final step before committing to fully rolling out a candidate model to production.
More on different types of testing here: Experiments and tests · Nextmv Docs