When performing a switchback test, can you test more than two models?
No. Switchback tests are designed to compare two models running at a time making operational decisions. Typically, if you want to compare many models side-by-side, you would run a batch experiment on historical data. Following that, you would likely run an acceptance test with the best model comparing it to your current state. When it’s time to run a live switchback test, that’s usually the last gate before fully launching a candidate model in place of the baseline model, and typically the question you’re asking is “am I safe to turn this model on in production when comparing it to my current model on live data”.