Data-Limited Continuous Experimentation (dlCE): A Literature Review

Stanislav Chren*, Fabian Fagerholm, Elizabeth Bjarnason, Johan Linåker, Saima Rafi, Bettina Lehtelä, Per Runeson, Marjo Kauppinen

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

Abstract

Continuous experimentation (CE) is a software development approach where product decisions are data-driven. Large global internet-facing companies such as Microsoft, Google, and Facebook apply the practice by leveraging their massive user bases to obtain high statistical significance in experimentation results. However, companies with smaller user bases, such as small- and medium-sized enterprises (SMEs) and early-stage software startups, struggle to adopt CE due to limitations in user data. Our goal is to increase understanding of situations in CE where data limitations occur, as observed in the research literature. We investigate data limitations and challenges related to them, including their characteristics, and solutions and practices that may address these. We conducted a rapid review of CE papers from a previous systematic literature review, and analysed these to identify scenarios that exhibit data limitations in CE. We present a framework that illustrates different dimensions of data-limited CE (DLC) and connects scenarios to challenges and their potential future solutions. Most challenges and potential solutions that we found are related to the amount of data. We also note examples of other limitations related to, e.g., evaluation metrics, that provide interesting avenues for further research.
Original languageEnglish
Title of host publicationProceedings of 15th International Conference on Software Business (ICSOB 2024)
PublisherSpringer
Number of pages15
Publication statusAccepted/In press - 2024
MoE publication typeA4 Conference publication
EventInternational Conference on Software Business - Utrecht, Netherlands
Duration: 18 Nov 202420 Nov 2024
Conference number: 15

Publication series

NameLecture notes in business information processing
Volume539
ISSN (Print)1865-1348
ISSN (Electronic)1865-1356

Conference

ConferenceInternational Conference on Software Business
Abbreviated titleICSOB
Country/TerritoryNetherlands
CityUtrecht
Period18/11/202420/11/2024

Fingerprint

Dive into the research topics of 'Data-Limited Continuous Experimentation (dlCE): A Literature Review'. Together they form a unique fingerprint.

Cite this