Calculate the expected impact of a new feature

Question

I would like to know how to calculate the estimated impact of the implementation of a new feature (on a website for example).

To put it in context, this is a case study, the first question being related to an AB test where we had to estimate the impact of a new technology, having the date of implementation of this technology, and having observations before and after (and therefore the setting up of control and test groups was easy).

For this second question, I was asked to estimate the impact of this new technology in the future, and therefore without having an implementation date.

How would you go about it? I have considered various methods such as randomized controlled trials (https://medium.com/data-science-at-microsoft/attribution-analysis-how-to-measure-impact-part-1-of-2-324d43fbbba0) but without much success.

Thanks in advance !

Answer 1

having observations before and after I'm afraid that something not easy to determine. You won't be able to isolate totally the effect of your new technology on the different metrics you're measuring (engagement, feature adoption, churn, time spent on platform etc.). Any seasonal effect, holidays, exceptional events, behavioral change, marketing effect, novelty effect could bias your measures.

The best way to measure the impact of a feature is to what we call an A/B test (running the experiment of controlled groups in parallel). Your sample of users are randomized (some A/B testing tools are using a randomization function for you) between group A and group B:

• Group A remains on the current version, we call it the "Control" group
• Group B receives the new version (your new feature), we call it the "Treatment" group

To estimate the impact of the feature in the future, that will be very inaccurate. You could either do some forecasting around your decision metrics (you'll to consider confidence intervals, margin errors etc.) or take another feature previously launched as a baseline and expect the same engagement around it.

Hope this helps