Candidate Solutions for Defining Explainability Requirements of AI Systems

[Context and Motivation] Many recent studies highlight explainability as an important requirement that supports in building transparent, trustworthy, and responsible AI systems. As a result, there is an increasing number of solutions that researchers have developed to assist in the definition of explainability requirements. [Question] We conducted a literature study to analyze what kind of candidate solutions are proposed for defining the explainability requirements of AI systems. The focus of this literature review is especially on the field of requirements engineering (RE). [Results] The proposed solutions for defining explainability requirements such as approaches, frameworks, and models are comprehensive. They can be used not only for RE activities but also for testing and evaluating the explainability of AI systems. In addition to the comprehensive solutions, we identified 30 practices that support the development of explainable AI systems. The literature study also revealed that most of the proposed solutions have not been evaluated in real projects, and there is a need for empirical studies. [Contribution] For researchers, the study provides an overview of the candidate solutions and describes research gaps. For practitioners, the paper summarizes potential practices that can help them define and evaluate the explainability requirements of AI systems.