Salma Imtiaz
ABSTRACT
Context: The use of Systematic Literature Review (SLR) requires expertise and poses many challenges for novice researchers. The experiences of those who have used this research methodology can benefit novice researchers in effectively dealing with these challenges. Objective: The aim of this study is to record the reported experiences of conducting Systematic Literature Reviews, for the benefit of new researchers. Such a review will greatly benefit the researchers wanting to conduct SLR for the very first time. Method: We conducted a tertiary study to gather the experiences published by researchers. Studies that have used the SLR research methodology in software engineering and have implicitly or explicitly reported their experiences are included in this review. Results: Our research has revealed 116 studies relevant to the theme. The data has been extracted by two researchers working independently and conflicts resolved after discussion with third researcher. Findings from these studies highlight Search Strategy, Online Databases, Planning and Data Extraction as the most challenging phases of SLR. Lack of standard terminology in software engineering papers, poor quality of abstracts and problems with search engines are some of the most cited challenges. Conclusion: Further research and guidelines is required to facilitate novice researchers in conducting these phases properly.
- Babar, M. A., and Zhang, H., 2009, Systematic Literature Reviews in Software Engineering: Preliminary Results from Interviews with Researchers, International Conference on Empirical Software Engineering and Measurement, pp. 346--355. Google Scholar
Digital Library
- Bailey, J., Zhang, C., Budgen, D., Charters, S., and Turner, M., 2007, Search Engine Overlaps: Do they Agree or Disagree?, Second International Workshop on Realizing Evidence-Based Software Engineering, p. 2--2. Google ScholarDigital Library
- Beecham, S., Baddoo, N., Hall, T., Robinson, H., and Sharp, H., 2006, Protocol for a Systematic Literature Review of Motivation in Software Engineering,.Google Scholar
- Brereton, O. P., Kitchenham, B. A., Budgen, D., Turner, M., and Khalil, M., 2007, Lessons from Applying the Systematic Literature Review Process within the Software Engineering Domain, Journal of Systems and Software, vol. 80, no. 4, pp. 571--583. Google ScholarDigital Library
- Cruzes, D. S and Dyba, T., 2011, Recommended Steps for Thematic Synthesis in Software Engineering, International Symposium on Empirical Software Engineering and Measurement, pp. 275--284. Google ScholarDigital Library
- Dieste, O., and Padua, O. A. G., 2007, Developing Search Strategies for Detecting Relevant Experiments for Systematic Reviews, Proceedings of First International Symposium on Empirical Software Engineering and Measurement, pp. 215--224. Google ScholarDigital Library
- Dybå, T., and Dingsøyr, T., 2008, Strength of Evidence in Systematic Reviews in Software Engineering, Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, pp. 178--187. Google ScholarDigital Library
- Dyba, T., Dingsoyr, T., Hanssen, G. K., and SINTEF, T., 2007, Applying Systematic Reviews to Diverse Study Types: An Experience Report, First International Symposium on Empirical Software Engineering and Measurement, 2007. pp. 225--234. Google ScholarDigital Library
- Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., and Linkman, S., 2009, Systematic Literature Reviews in Software Engineering-A Systematic Literature Review, Information and Software Technology, vol. 51, no. 1, pp. 7--15. Google ScholarDigital Library
- Kitchenham, B. A., Budgen, D. and Brereton, O. P., 2006, Evidence-based Software Engineering and Systematic Literature Reviews, Product-Focused Software Process Improvement, p. 3--3, DOI=10.1007/11767718_3. Google ScholarDigital Library
- Kitchenham, B., and Charters, S., 2007, Guidelines for performing Systematic Literature Reviews in Software Engineering', Technical Report EBSE 2007--001, Keele University and Durham University Joint Report.Google Scholar
- Kitchenham, B. A., Dyba, T., and Jorgensen, M., 2004, Evidence-Based Software Engineering, Proceedings of 26th International Conference on Software Engineering, pp. 273--281. Google ScholarDigital Library
- Kitchenham, B., Pretorius, R., Budgen, D., Brereton, O. P., Turner, M., Niazi, M., and Linkman, S., 2010, Systematic Literature Reviews in Software Engineering--A Tertiary Study, Information and Software Technology, vol. 52, no. 8, pp. 792--805. Google ScholarDigital Library
- Riaz, M., Sulayman, M., Salleh, N., and Mendes, E., 2010, Experiences Conducting Systematic Reviews from Novices' Perspective, Proceedings of 14th International Conference on Evaluation and Assessment in Software Engineering (EASE). Google ScholarDigital Library
- Staples, M. and Niazi, M., 2007, Experiences using systematic review guidelines, Journal of Systems and Software, vol. 80, no. 9, pp. 1425--1437. Google ScholarDigital Library
- Silva, F. Q. B., Santos, A. L. M., Soares, S., França, A. C. C., Monteiro, C. V. F., and Maciel, F. F., 2011, Six years of Systematic Literature Reviews in Software Engineering: An Updated Tertiary Study, Information and Software Technology, vol. 53, no. 9, pp. 899--913. Google ScholarDigital Library
- MacDonell, S., Shepperd, M., Kitchenham B. A., and Mendes, E., How reliable are systematic reviews in empirical software engineering?, IEEE Transactions on Software Engineering, 2010, vol. 30, no. 5, pp. 676--687. Google ScholarDigital Library
Index Terms
- A tertiary study: experiences of conducting systematic literature reviews in software engineering
Recommendations
Experience-based guidelines for effective and efficient data extraction in systematic reviews in software engineering
EASE '17: Proceedings of the 21st International Conference on Evaluation and Assessment in Software EngineeringTo systematically collect evidence and to structure a given area in software engineering (SE), Systematic Literature Reviews (SLR) and Systematic Mapping (SM) studies have become common. Data extraction is one of the main phases (activities) when ...
The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature
EASE '16: Proceedings of the 20th International Conference on Evaluation and Assessment in Software EngineeringSystematic Literature Reviews (SLR) may not provide insight into the "state of the practice" in SE, as they do not typically include the "grey" (non-published) literature. A Multivocal Literature Review (MLR) is a form of a SLR which includes grey ...
Systematic literature reviews in software engineering - A tertiary study
Context: In a previous study, we reported on a systematic literature review (SLR), based on a manual search of 13 journals and conferences undertaken in the period 1st January 2004 to 30th June 2007. Objective: The aim of this on-going research is to ...
Comments