WASP Continuous Experimentation with Limited User Data

Project Details

Description

Continuous experimentation (CE) uses field experiments to steer software product development. This practice is common in global internet-based companies with large user bases and the ability to collect huge volumes of user data from widely deployed products. In contrast, this project extends CE to contexts with limited amounts of user data.

The aim of this project is to explore and design solutions for CE when developing products for which there are limited amounts of user data and to investigate use of different types of user data (qualitative vs. quantitative). We will focus on products of differing maturity from ideation (early-stage software development) towards growth (B2B applications and in-house systems & tools). We will also design novel methods for obtaining and analysing user data in these contexts to enable data-limited CE.

The goal is to enable companies with limited amounts of user data to gain from applying CE and thereby obtain improvements to user satisfaction and problem-solution fit. More companies can then be supported in designing “the right software” (validation through CE) rather than just designing the software right (verification).

Our research targets the problem domain and novel solutions for data-limited CE; alternative experimentation setups and statistical methods (e.g. Bayesian approaches); qualitative metrics (based e.g. on interviews, questionnaires, and prototype testing); and ways of reusing existing user bases of other products (e.g. piggybacking (Greve 2015), proxy users). Our main research questions are:
RQ1 What relations are there to product and business maturity for data-limited CE?
- RQ1.1 How can CE be applied to products in ideation and subsequent stages?
- RQ1.2 How can data-limited CE be used to support timely strategic and operational decisions?

RQ2 How can CE be applied when there are limited amounts of user data?
- RQ2.1 What statistical methods and approaches are suitable?
- RQ2.2 How can a combination of qualitative and quantitative data be useful?
- RQ2.3 How can existing user bases be (re)used?

Research approach
The project will produce prescriptive knowledge to support professionals in applying data-limited CE. Through a Design Science (DS) approach (Runeson 2020), we will design solutions to real world problems, gain empirical insights from industrial case studies, and evaluate these solutions in context through pilot studies. Knowledge will be produced through an iterative process of problem exploration (RQ1), solution design (RQ2), and evaluation for two problem contexts. Each site will focus on one context, namely early-stage (Aalto) and in-house software development (Lund), and will share findings with the other and collaborate on solution designs. This approach strengthens the empirical base of our work and enables a broader evaluation of solutions by pooling industrial contacts and combining insights from multiple industrial cases.

Through industrial case studies and pilot studies, we will explore the problem of data-limited CE (RQ1) and design novel solutions. Our solution designs will address the areas outlined in RQ2.1-3, namely statistical methods and experimental set-up for smaller volumes of data, combining quantitative and qualitative metrics, and how to re-use existing user bases in CE when developing new systems.

Our research will be based on existing theoretical foundations for the field and will contribute to the state-of-the art. In particular, our work is based on Fagerholm’s seminal RIGHT model for CE (2017), the work by WASP PhD student Ros (2022), and Bjarnason’s (2021ab) studies of prototyping in software startups. In addition, the conceptual framework developed by Kauppinen's research team (Töhönen et al. 2020) provides a basis for evaluating experiment results from a business perspective. The research findings will be published in high-quality scientific journals and conferences and consist of empirical knowledge of the problem domain and evaluated solutions for data-limited CE.

References
Bjarnason, E. (2021a), Prototyping Practices in Software Startups: Initial Case Study Results, 29th Int Reqts Engin Conf Workshops (REW), 2021, pp. 206-211, doi: 10.1109/REW53955.2021.00038.

Bjarnason, E., Lang, F., & Mjöberg, A. (2021b). A Model of Software Prototyping based on a Systematic Map. In Pro of 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2021, pp. 1-11.

Fagerholm, F., Sanchez Guinea, Mäenpää, and Münch. (2017) The RIGHT Model for Continuous Experimentation. Journal of Systems and Software, 123:292–305, 2017.

Grevet, C., & Gilbert, E. (2015). Piggyback prototyping: Using existing, large-scale social computing systems to prototype new ones. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 4047-4056).

Ros R. (2022) Understanding and Improving Continuous Experimentation From A/B Testing to Continuous Software Optimization, PhD thesis, Lund University.

Runeson, P., E. Engström, and M.-A. Storey. (2020) The design science paradigm as a frame for empirical software engineering. In M. Felderer and G. H. Travassos, editors, Contemporary Empirical Methods in Software Engineering. Springer, 2020.

Töhönen, H., Kauppinen, M., Männistö, T., and Itälä, T., (2020). A conceptual framework for valuing IT within a business system. International Journal of Accounting Information Systems, 36:100442.

Layman's description

Continuous experimentation (CE) uses field experiments to steer software product development. This practice is common in global internet-based companies with large user bases and the ability to collect huge volumes of user data from widely deployed products. In contrast, this project extends CE to contexts with limited amounts of user data.
Short titleData-limited CE
StatusFinished
Effective start/end date01/10/202201/10/2024

Keywords

  • continuous experimentation
  • software engineering

Fingerprint

Explore the research topics touched on by this project. These labels are generated based on the underlying awards/grants. Together they form a unique fingerprint.