Systematic literature reviews in software engineering – A systematic literature review

https://doi.org/10.1016/j.infsof.2008.09.009Get rights and content

Abstract

Background

In 2004 the concept of evidence-based software engineering (EBSE) was introduced at the ICSE04 conference.

Aims

This study assesses the impact of systematic literature reviews (SLRs) which are the recommended EBSE method for aggregating evidence.

Method

We used the standard systematic literature review method employing a manual search of 10 journals and 4 conference proceedings.

Results

Of 20 relevant studies, eight addressed research trends rather than technique evaluation. Seven SLRs addressed cost estimation. The quality of SLRs was fair with only three scoring less than 2 out of 4.

Conclusions

Currently, the topic areas covered by SLRs are limited. European researchers, particularly those at the Simula Laboratory appear to be the leading exponents of systematic literature reviews. The series of cost estimation SLRs demonstrate the potential value of EBSE for synthesising evidence and making it available to practitioners.

Introduction

At ICSE04, Kitchenham et al. [23] suggested software engineering researchers should adopt “Evidence-based Software Engineering” (EBSE). EBSE aims to apply an evidence-based approach to software engineering research and practice. The ICSE paper was followed-up by an article in IEEE Software [5] and a paper at Metrics05 [17].

Evidence-based research and practice was developed initially in medicine because research indicated that expert opinion based medical advice was not as reliable as advice based on the accumulation of results from scientific experiments. Since then many domains have adopted this approach, e.g. Criminology, Social policy, Economics, Nursing etc. Based on Evidence-based medicine, the goal of Evidence-based Software Engineering is:

“To provide the means by which current best evidence from research can be integrated with practical experience and human values in the decision making process regarding the development and maintenance of software” [5].

In this context, evidence is defined as a synthesis of best quality scientific studies on a specific topic or research question. The main method of synthesis is a systematic literature review (SLR). In contrast to an expert review using ad hoc literature selection, an SLR is a methodologically rigorous review of research results. The aim of an SLR is not just to aggregate all existing evidence on a research question; it is also intended to support the development of evidence-based guidelines for practitioners. The end point of EBSE is for practitioners to use the guidelines to provide appropriate software engineering solutions in a specific context.

The purpose of this study is to review the current status of EBSE since 2004 using a tertiary study to review articles related to EBSE and, in particular, we concentrate on articles describing systematic literature reviews (SLRs). Although SLRs are not synonymous with EBSE, the aggregation of research results is an important part of the EBSE process and, furthermore, is the part of the EBSE process that can be readily observed in the scientific literature. We describe our methodology in Section 2 and present our results in Section 3. In Section 4 we answer our 4 major research questions. We present our conclusions in Section 5.

Section snippets

Method

This study has been undertaken as a systematic literature review based on the original guidelines as proposed by Kitchenham [22]. In this case the goal of the review is to assess systematic literature reviews (which are referred to as secondary studies), so this study is categorised as a tertiary literature review. The steps in the systematic literature review method are documented below.

Results

This section summarizes the results of the study.

Discussion

In this section, we discuss the answers to our research questions.

Conclusions

Although 10 of the SLR studies in this review cited one of the EBSE papers [5] or the SLR Guidelines [22], the number of SLRs has remained extremely stable in the 3.5 years included in this study. Furthermore, Table A2 (see Appendix 1) also makes it clear that many researchers still prefer to undertake informal literature surveys. However, we have found that the quality of SLRs is improving, suggesting that researchers who are interested in the EBSE approach are becoming more competent in the

Acknowledgements

This research was funded by The Engineering and Physical Sciences Research Council (EPSRC) EBSE Project (EP/C51839X/1). Short, preliminary versions of this study were presented at the REBSE2 workshop at ICSE07 and the EASE07 Conference at Keele University.

References (42)

  • M. Zelkowitz et al.

    Experimental validation in software engineering

    Information and Software Technology

    (1997)
  • R.F. Barcelos, G.H. Travassos, Evaluation approaches for software architectural documents: a systematic review, in:...
  • O.P. Brereton et al.

    Lessons from applying the systematic literature review process within the software engineering domain

    Journal of Systems and Software

    (2007)
  • Centre for Reviews and Dissemination, What are the criteria for the inclusion of reviews on DARE? 2007. Available at...
  • T. Dybå et al.

    Evidence-based software engineering for practitioners

    IEEE Software

    (2005)
  • M. Dyer et al.

    Systematic Reviews in Evidence-Based Software Technology and Software Engineering

    (2005)
  • D. Galin et al.

    Do SQA programs work – CMM works. A meta analysis

    IEEE International Conference on Software – Science, Technology and Engineering

    (2005)
  • D. Galin et al.

    Are CMM program investments beneficial? Analyzing past studies

    IEEE Software

    (2006)
  • R.L. Glass et al.

    An analysis of research in computing disciplines

    CACM

    (2004)
  • J.E. Hannay et al.

    A systematic review of theory use in software engineering experiments

    IEEE Transactions on SE

    (2007)
  • W. Hayes

    Research synthesis in software engineering: the case for meta-analysis

    (1999)
  • Cited by (2742)

    • VALIDATE: A deep dive into vulnerability prediction datasets

      2024, Information and Software Technology
    • Supporting reusable model migration with Edelta

      2024, Journal of Systems and Software
    View all citing articles on Scopus
    View full text