Data-Limited Continuous Experimentation (dlCE): A Literature Review

Continuous experimentation (CE) is a software development approach where product decisions are data-driven. Large global internet-facing companies such as Microsoft, Google, and Facebook apply the practice by leveraging their massive user bases to obtain high statistical significance in experimentation results. However, companies with smaller user bases, such as small- and medium-sized enterprises (SMEs) and early-stage software startups, struggle to adopt CE due to limitations in user data. Our goal is to increase understanding of situations in CE where data limitations occur, as observed in the research literature. We investigate data limitations and challenges related to them, including their characteristics, and solutions and practices that may address these. We conducted a rapid review of CE papers from a previous systematic literature review, and analysed these to identify scenarios that exhibit data limitations in CE. We present a framework that illustrates different dimensions of data-limited CE (DLC) and connects scenarios to challenges and their potential future solutions. Most challenges and potential solutions that we found are related to the amount of data. We also note examples of other limitations related to, e.g., evaluation metrics, that provide interesting avenues for further research.