Systematic literature reviews in software engineering – A systematic literature review
Introduction
At ICSE04, Kitchenham et al. [23] suggested software engineering researchers should adopt “Evidence-based Software Engineering” (EBSE). EBSE aims to apply an evidence-based approach to software engineering research and practice. The ICSE paper was followed-up by an article in IEEE Software [5] and a paper at Metrics05 [17].
Evidence-based research and practice was developed initially in medicine because research indicated that expert opinion based medical advice was not as reliable as advice based on the accumulation of results from scientific experiments. Since then many domains have adopted this approach, e.g. Criminology, Social policy, Economics, Nursing etc. Based on Evidence-based medicine, the goal of Evidence-based Software Engineering is:
“To provide the means by which current best evidence from research can be integrated with practical experience and human values in the decision making process regarding the development and maintenance of software” [5].
In this context, evidence is defined as a synthesis of best quality scientific studies on a specific topic or research question. The main method of synthesis is a systematic literature review (SLR). In contrast to an expert review using ad hoc literature selection, an SLR is a methodologically rigorous review of research results. The aim of an SLR is not just to aggregate all existing evidence on a research question; it is also intended to support the development of evidence-based guidelines for practitioners. The end point of EBSE is for practitioners to use the guidelines to provide appropriate software engineering solutions in a specific context.
The purpose of this study is to review the current status of EBSE since 2004 using a tertiary study to review articles related to EBSE and, in particular, we concentrate on articles describing systematic literature reviews (SLRs). Although SLRs are not synonymous with EBSE, the aggregation of research results is an important part of the EBSE process and, furthermore, is the part of the EBSE process that can be readily observed in the scientific literature. We describe our methodology in Section 2 and present our results in Section 3. In Section 4 we answer our 4 major research questions. We present our conclusions in Section 5.
Section snippets
Method
This study has been undertaken as a systematic literature review based on the original guidelines as proposed by Kitchenham [22]. In this case the goal of the review is to assess systematic literature reviews (which are referred to as secondary studies), so this study is categorised as a tertiary literature review. The steps in the systematic literature review method are documented below.
Results
This section summarizes the results of the study.
Discussion
In this section, we discuss the answers to our research questions.
Conclusions
Although 10 of the SLR studies in this review cited one of the EBSE papers [5] or the SLR Guidelines [22], the number of SLRs has remained extremely stable in the 3.5 years included in this study. Furthermore, Table A2 (see Appendix 1) also makes it clear that many researchers still prefer to undertake informal literature surveys. However, we have found that the quality of SLRs is improving, suggesting that researchers who are interested in the EBSE approach are becoming more competent in the
Acknowledgements
This research was funded by The Engineering and Physical Sciences Research Council (EPSRC) EBSE Project (EP/C51839X/1). Short, preliminary versions of this study were presented at the REBSE2 workshop at ICSE07 and the EASE07 Conference at Keele University.
References (42)
- et al.
A systematic review of statistical power in software engineering experiments
Information and Software Technology
(2006) - et al.
Research in software engineering: an analysis of the literature
Information and Software technology
(2002) - et al.
Software effort estimation terminology: the tower of Babel
Information and Software Technology
(2006) Estimation of software development work effort: evidence on expert judgement and formal models
International Journal of Forecasting
(2007)A review of studies on expert estimation of software development effort
Journal of Systems and Software
(2004)Applying meta-analytical procedures to software engineering experiments
JSS
(2000)- et al.
Capture–recapture in software inspections after 10 years research – theory, evaluation and application
Journal of Systems and Software
(2004) - et al.
Combining empirical results in software engineering
Information and Software Technology
(1998) - et al.
Research in computer science: an empirical study
Journal of Systems and Software
(2004) - et al.
Experimental evaluation in computer science: a quantitative study
Journal of Systems and Software
(1995)
Experimental validation in software engineering
Information and Software Technology
Lessons from applying the systematic literature review process within the software engineering domain
Journal of Systems and Software
Evidence-based software engineering for practitioners
IEEE Software
Systematic Reviews in Evidence-Based Software Technology and Software Engineering
Do SQA programs work – CMM works. A meta analysis
IEEE International Conference on Software – Science, Technology and Engineering
Are CMM program investments beneficial? Analyzing past studies
IEEE Software
An analysis of research in computing disciplines
CACM
A systematic review of theory use in software engineering experiments
IEEE Transactions on SE
Research synthesis in software engineering: the case for meta-analysis
Cited by (2742)
VALIDATE: A deep dive into vulnerability prediction datasets
2024, Information and Software TechnologySupporting reusable model migration with Edelta
2024, Journal of Systems and SoftwareTowards an integrated framework for developing blockchain systems
2024, Decision Support SystemsModel driven engineering for machine learning components: A systematic literature review
2024, Information and Software TechnologyTowards efficient irrigation management at field scale using new technologies: A systematic literature review
2024, Agricultural Water ManagementBlockchain technology meets 6 G wireless networks: A systematic survey
2024, Alexandria Engineering Journal