How many times have you had the “but is it significant” fight when you try to analyze an experiment? How confident do you feel that you’re making good calls? Are you worried the statistics police are going to knock on your door if you look at your results too early? But how can you not look if the experiment may be costing you money?
We had the same questions and we’ve found some techniques that work for us. Borrowing liberally from AirBnB we now use “p-value graphs” for all our tests and our comfort level has increased significantly.
