Skip to main content

Advertisement

SpringerLink
Log in

Teaching requirements elicitation interviews: an empirical study of learning from mistakes

  • RE 2018
  • Published:
Requirements Engineering Aims and scope Submit manuscript

Abstract

Interviews are the most widely used elicitation technique in requirements engineering (RE). However, conducting a requirements elicitation interview is challenging. The mistakes made in design or conduct of the interviews can create problems in the later stages of requirements analysis. Empirical evidence about effective pedagogical approaches for training novices on conducting requirements elicitation interviews is scarce. In this paper, we present a novel pedagogical approach for training student analysts in the art of elicitation interviews. Our study is conducted in two parts: first, we perform an observational study of interviews performed by novices, and we present a classification of the most common mistakes made; second, we utilize this list of mistakes and monitor the students’ progress in three set of interviews to discover the individual areas for improvement. We conducted an empirical study involving role-playing and authentic assessment in two semesters on two different cohorts of students. In the first semester, we had 110 students, teamed up in 28 groups, to conduct three interviews with stakeholders. We qualitatively analysed the data to identify and classify the mistakes made from their first interview only. In the second semester, we had 138 students in 34 groups and we monitored and analysed their progress in all three interviews by utilizing the list of mistakes from the first study. First, we identified 34 unique mistakes classified into seven high-level themes, namely question formulation, question omission, interview order, communication skills, analyst behaviour, customer interaction, teamwork and planning. In the second study, we discovered that the students struggled mostly in the areas of question formulation, question omission and interview order and did not manage to improve their skills throughout the three interviews. Our study presents a novel and repeatable pedagogical design, and our findings extend the body of knowledge aimed at RE education and training by providing an empirically grounded categorization of mistakes made by novices. We offer an analysis of the main pain points in which instructors should pay more attention during their design and training.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Notes

  1. Case Study 1: https://drive.google.com/file/d/1DLXjLUnqISrq2XlSmuZYf4GlS9FGhuZP/view?usp=sharing.

    Case Study 2: https://drive.google.com/file/d/1EMOhQOw4GCFfQoT7nCYgOlSN4sLNain-/view?usp=sharing.

  2. The authors playing the role of customer and observer were not available in study 2 to have an active role during the class term; therefore, we do not have customer think aloud or observation data in study 2.

References

  1. Briggs CL (1986) Learning how to ask: a sociolinguistic appraisal of the role of the interview in social science research, vol 1. Cambridge University Press, Cambridge

    Book  Google Scholar 

  2. Zowghi D, Coulin C (2005) Requirements elicitation: a survey of techniques, approaches, and tools. In: Engineering and managing software requirements. Springer, pp 19–46

  3. Sutcliffe A, Sawyer P (2013) Requirements elicitation: towards the unknown unknowns. In: 2013 21st IEEE international on requirements engineering conference (RE). IEEE

  4. Davis A et al (2006) Effectiveness of requirements elicitation techniques: empirical results derived from a systematic review. In: 14th IEEE international conference on requirements engineering. IEEE

  5. Dieste O, Juristo N (2011) Systematic review and aggregation of empirical studies on elicitation techniques. IEEE Trans Softw Eng 37(2):283–304

    Article  Google Scholar 

  6. Donati B et al (2017) Common mistakes of student analysts in requirements elicitation interviews. In: International working conference on requirements engineering: foundation for software quality. Springer

  7. Pitts MG, Browne GJ (2007) Improving requirements elicitation: an empirical investigation of procedural prompts. Inf Syst J 17(1):89–110

    Article  Google Scholar 

  8. Hogarth RM et al (1991) Learning from feedback: exactingness and incentives. J Exp Psychol Learn Mem Cogn 17(4):734

    Article  Google Scholar 

  9. Li S (2010) The effectiveness of corrective feedback in SLA: a meta-analysis. Lang Learn 60(2):309–365

    Article  Google Scholar 

  10. Svensson RB, Regnell B (2017) Is role playing in Requirements Engineering Education increasing learning outcome? Requir Eng 22(4):475–489

    Article  Google Scholar 

  11. Zowghi D, Paryani S (2003) Teaching requirements engineering through role playing: lessons learnt. In: 11th IEEE international on requirements engineering conference. Proceedings. IEEE

  12. Yusop N, Mehboob Z, Zowghi D (2007) The role of conducting stakeholder meeting in requirements engineering techniques. In: International workshop on the requirements engineering education and training. IEEE Computer Society

  13. Bano M et al (2018) Learning from mistakes: an empirical study of elicitation interviews performed by novices. In: 2018 IEEE 26th international requirements engineering conference (RE). IEEE

  14. Spoletini P, Ferrari A (2017) Requirements elicitation: a look at the future through the lenses of the past. In: 2017 IEEE 25th international on requirements engineering conference (RE). IEEE

  15. Aranda AM, Dieste O, Juristo N (2016) Effect of domain knowledge on elicitation effectiveness: an internally replicated controlled experiment. IEEE Trans Software Eng 42(5):427–451

    Article  Google Scholar 

  16. Hadar I, Soffer P, Kenzi K (2014) The role of domain knowledge in requirements elicitation via interviews: an exploratory study. Requir Eng 19(2):143–159

    Article  Google Scholar 

  17. Niknafs A, Berry D (2017) The impact of domain knowledge on the effectiveness of requirements engineering activities. Empir Softw Eng 22(1):80–133

    Article  Google Scholar 

  18. Niknafs A, Berry DM (2013) An industrial case study of the impact of domain ignorance on the effectiveness of requirements idea generation during requirements elicitation. In: 2013 21st IEEE international on requirements engineering conference (RE). IEEE

  19. Pitts MG, Browne GJ (2004) Stopping behavior of systems analysts during information requirements elicitation. J Manag Inf Syst 21(1):203–226

    Article  Google Scholar 

  20. Distanont A et al (2012) The engagement between knowledge transfer and requirements engineering. Int J Manag Knowl Learn 1(2):131–156

    Google Scholar 

  21. Ferrari A, Spoletini P, Gnesi S (2016) Ambiguity and tacit knowledge in requirements elicitation interviews. Requir Eng 21(3):333–355

    Article  Google Scholar 

  22. Coughlan J, Macredie RD (2002) Effective communication in requirements elicitation: a comparison of methodologies. Requir Eng 7(2):47–60

    Article  Google Scholar 

  23. Agarwal R, Tanniru MR (1990) Knowledge acquisition using structured interviewing: an empirical investigation. J Manag Inf Syst 7(1):123–140

    Article  Google Scholar 

  24. Browne GJ, Rogich MB (2001) An empirical investigation of user requirements elicitation: comparing the effectiveness of prompting techniques. J Manag Inf Syst 17(4):223–249

    Article  Google Scholar 

  25. Shuraida S, Barki H (2013) The influence of analyst communication in IS projects. J Assoc Inf Syst 14(9):482

    Google Scholar 

  26. Portugal S (2013) Interviewing users: how to uncover compelling details. Louis Rosenfeld, New York

    Google Scholar 

  27. Lynda.com, Requirements elicitation for business analysts: interviews. https://www.lynda.com/Communication-tutorials/Requirements-Elicitation-Interviews/410330-2.html. Accessed 14 May 2019

  28. International, L.T., Developing user requirements: the key to project success. https://www.learningtree.com/courses/315/developing-user-requirements-training-the-key-to-project-success/. Accessed 14 May 2019

  29. Walcott-Justice K Requirements elicitation: artifact and stakeholder analysis. Coursera.com. https://www.coursera.org/learn/requirements-elicitation/lecture/FAN2U/stakeholder-elicitation-starting-interviewing-techniques. Accessed 14 May 2019

  30. Hathaway T, Hathaway A (2016) Requirements elicitation interviews and workshops—simply put!: best practices, skills, and attitudes for requirements gathering on IT projects. BA-Experts, USA

    Google Scholar 

  31. Adams S (2001) Interviewing for journalists. Psychology Press, London

    Google Scholar 

  32. Martin JR (2017) Actuality interviewing and listening: how to conduct successful interviews for nonfiction storytelling, actuality documentaries and other disciplines. Real Deal Press, USA

    Google Scholar 

  33. Grobel L (2010) The art of the interview: lessons from a master of the craft. Three Rivers Press, NY, USA

    Google Scholar 

  34. De Burgh H (2003) Skills are not enough: the case for journalism as an academic discipline. Journalism 4(1):95–112

    Article  Google Scholar 

  35. DiCicco-Bloom B, Crabtree BF (2006) The qualitative research interview. Med Educ 40(4):314–321

    Article  Google Scholar 

  36. Brinkmann S (2014) Interview. In: Encyclopedia of critical psychology. Springer, pp 1008–1010

  37. Jacob SA, Furgerson SP (2012) Writing interview protocols and conducting interviews: tips for students new to the field of qualitative research. Qual Rep 17(42):1–10

    Google Scholar 

  38. Turner DI (2010) Qualitative interview design: a practical guide for novice investigators. Qual Rep 15(3):754

    Google Scholar 

  39. Dilley P (2000) Conducting successful interviews: tips for intrepid research. Theory Pract 39(3):131–137

    Article  Google Scholar 

  40. Seidman I (2013) Interviewing as qualitative research: a guide for researchers in education and the social sciences. Teachers College Press, New York

    Google Scholar 

  41. Ritchie J et al (2013) Qualitative research practice: a guide for social science students and researchers. Sage, Thousand Oaks

    Google Scholar 

  42. Morrison J (2014) The first interview. Guilford Publications, New York

    Google Scholar 

  43. Warner RE (2013) Solution-focused interviewing: applying positive psychology, a manual for practitioners. University of Toronto Press, Toronto

    Google Scholar 

  44. Miller C (2003) Interviewing strategies. In: Diagnostic interviewing. Springer, pp 47–66

  45. Hoffman CD (2005) Investigative interviewing: strategies and techniques. International Foundation for Protection Officers, Naples

    Google Scholar 

  46. Investigations, I.f.I.C. (2017) Investigative interview skills course. https://iici.global/course/investigative-interview-skills-course. Accessed 14 May 2019

  47. Navarro EO (2011) On the role of learning theories in furthering software engineering education. In: Instructional design: concepts, methodologies, tools and applications. IGI Global, pp 1645–1666

  48. Dewey J (1916) Education and democracy. Macmillan, New York

    Google Scholar 

  49. Lave J (1988) Cognition in practice: mind, mathematics and culture in everyday life. Cambridge University Press, Cambridge

    Book  Google Scholar 

  50. Bruner JS (1979) On knowing: Essays for the left hand. Harvard University Press, Cambridge

    Google Scholar 

  51. Schank R (1997) Virtual learning. A revolutionary approach to building a highly skilled workforce. ERIC, New York

    Google Scholar 

  52. Schön DA (1987) Educating the reflective practitioner. Jossey-Bass, San Francisco

    Google Scholar 

  53. Moore M, Potts C (1994) Learning by doing: goals and experiences of two software engineering project courses. In: Conference on software engineering education. Springer

  54. Tvedt JD, Tesoriero R, Gary KA (2001) The software factory: combining undergraduate computer science and software engineering education. In: Proceedings of the 23rd international conference on software engineering. IEEE Computer Society

  55. Germain T, Robillard PN, Dulipovici M (2002) Process activities in a project based course in software engineering. In: Frontiers in education. FIE 2002. 32nd Annual. IEEE

  56. dos Santos SC, Soares FS (2013) Authentic assessment in software engineering education based on PBL principles: a case study in the telecom market. In: Proceedings of the 2013 international conference on software engineering. IEEE Press

  57. Herrington J, Herrington A (1998) Authentic assessment and multimedia: how university students respond to a model of authentic assessment. Higher Educ Res Dev 17(3):305–322

    Article  Google Scholar 

  58. Gulikers JT, Bastiaens TJ, Kirschner PA (2004) A five-dimensional framework for authentic assessment. Educ Tech Res Dev 52(3):67

    Article  Google Scholar 

  59. Dawson R (2000) Twenty dirty tricks to train software engineers. In: Proceedings of the 22nd international conference on Software engineering. ACM

  60. Ferrari A et al (2017) Interview review: detecting latent ambiguities to improve the requirements elicitation process. In: 2017 IEEE 25th international on requirements engineering conference (RE). IEEE

  61. Spoletini P et al (2018) Interview review: an empirical study on detecting ambiguities in requirements elicitation interviews. In: International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer

  62. Burnay C, Jureta IJ, Faulkner S (2014) What stakeholders will or will not say: a theoretical and empirical study of topic importance in Requirements Engineering elicitation interviews. Inf Syst 46:61–81

    Article  Google Scholar 

  63. Moody JW, Blanton JE, Cheney PH (1998) A theoretically grounded approach to assist memory recall during information requirements determination. J Manag Inf Syst 15(1):79–98

    Article  Google Scholar 

  64. Wetherbe JC (1991) Executive information requirements: getting it right. Mis Q 15:51–65

    Article  Google Scholar 

  65. Pacheco C, Garcia I (2012) A systematic literature review of stakeholder identification methods in requirements elicitation. J Syst Softw 85(9):2171–2181

    Article  Google Scholar 

  66. Gervasi V et al (2013) Unpacking tacit knowledge for requirements engineering. In: Managing requirements knowledge. Springer, pp 23–47

  67. Saiedian H, Dale R (2000) Requirements engineering: making the connection between the software developer and customer. Inf Softw Technol 42(6):419–428

    Article  Google Scholar 

  68. Lauer TW, Peacock E, Jacobs SM (1992) Question generation and the systems analysis process. In: Questions and information systems, pp 47–61

  69. Coughlan J, Lycett M, Macredie RD (2003) Communication issues in requirements elicitation: a content analysis of stakeholder experiences. Inf Softw Technol 45(8):525–537

    Article  Google Scholar 

  70. Gallivan MJ, Keil M (2003) The user–developer communication process: a critical case study. Inf Syst J 13(1):37–68

    Article  Google Scholar 

  71. Berry DM (1995) The importance of ignorance in requirements engineering. J Syst Softw 28:179–184

    Article  Google Scholar 

  72. Denzin NK, Lincoln YS (1994) Handbook of qualitative research. Sage Publications Inc., Thousand Oaks

    Google Scholar 

  73. Karras O, Kiesling S, Schneider K (2016) Supporting requirements elicitation by tool-supported video analysis. In: 2016 IEEE 24th international on requirements engineering conference (RE). IEEE

Download references

Acknowledgements

Authors would like to thank all the students who participated in this project. This research was approved by the University of Technology Sydney’s Research Ethics committee, under the number ETH17-1266. This work was partially supported by the National Science Foundation under grant CCF-1718377.

Author information

Authors and Affiliations

  1. Swinburne University of Technology, Melbourne, Australia

    Muneera Bano

  2. University of Technology Sydney, Sydney, Australia

    Didar Zowghi

  3. CNR-ISTI, Pisa, Italy

    Alessio Ferrari

  4. Kennesaw State University, Kennesaw, GA, USA

    Paola Spoletini

  5. DILEF, University of Florence, Florence, Italy

    Beatrice Donati

Authors
  1. Muneera Bano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Didar Zowghi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alessio Ferrari
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Paola Spoletini
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Beatrice Donati
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muneera Bano.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1: Interview Questionnaire

Please rate your agreement with the following statements about QUESTION FORMULATION * [Strongly Agree (1), Agree (2), Not sure (3), Disagree (4), Strongly Disagree (5)]

The analyst asked vague questions

The analyst asked technical questions

The analyst asked questions that appeared irrelevant to me

The analyst asked the customer for solutions

The analyst asked long and overly complex questions

The analyst formulated their questions in a way that appeared incorrect to me

The analyst asked vague questions

The analyst asked technical questions

The analyst asked questions that appeared irrelevant to me

The analyst asked the customer for solutions

The analyst asked long and overly complex questions

The analyst formulated their questions in a way that appeared incorrect to me

Please rate your agreement with the following statements about QUESTION OMISSION * [Strongly Agree (1), Agree (2), Not sure (3), Disagree (4), Strongly Disagree (5)]

The analyst DID NOT ask for additional stakeholders

The analyst DID NOT ask probing questions to confirm their understanding

The analyst DID NOT ask about the existing system or business process

The analyst DID NOT ask questions about feature prioritisation

The analyst DID NOT ask information about the problem domain

The analyst DID NOT identify goals and success criteria

The analyst DID NOT ask all the questions that I consider relevant

The analyst DID NOT ask for additional stakeholders

The analyst DID NOT ask probing questions to confirm their understanding

The analyst DID NOT ask about the existing system or business process

The analyst DID NOT ask questions about feature prioritisation

The analyst DID NOT ask information about the problem domain

The analyst DID NOT identify goals and success criteria

The analyst DID NOT ask all the questions that I consider relevant

Please rate your agreement with the following statements about ORDER OF INTERVIEW * [Strongly Agree (1), Agree (2), Not sure (3), Disagree (4), Strongly Disagree (5)]

The analyst DID NOT perform a summary at the end of the interview

The analyst started the interview by asking direct questions about the system

The analyst asked questions in an order that appeared incorrect to me

The analyst repeated the same questions multiple times

The analyst DID NOT perform a summary at the end of the interview

The analyst started the interview by asking direct questions about the system

The analyst asked questions in an order that appeared incorrect to me

The analyst repeated the same questions multiple times

Please rate your agreement with the following statements about COMMUNICATION SKILLS * [Strongly Agree (1), Agree (2), Not sure (3), Disagree (4), Strongly Disagree (5)]

The dialogues style used by the analyst appears unnatural to me

The analyst showed poor communication skills

The analyst showed poor listening skills

The analyst spoke with a low and unclear tone

The dialogues style used by the analyst appears unnatural to me

The analyst showed poor communication skills

The analyst showed poor listening skills

The analyst spoke with a low and unclear tone

Please rate your agreement with the following statements about ANALYST BEHAVIOUR * [Strongly Agree (1), Agree (2), Not sure (3), Disagree (4), Strongly Disagree (5)]

The analyst showed lack of confidence

The analyst appeared overconfident or arrogant

The analyst showed a passive attitude

The analyst showed a behaviour that appeared unprofessional to me

The analyst showed lack of confidence

The analyst appeared overconfident or arrogant

The analyst showed a passive attitude

The analyst showed a behaviour that appeared unprofessional to me

Please rate your agreement with the following statements about CUSTOMER INTERACTION * [Strongly Agree (1), Agree (2), Not sure (3), Disagree (4), Strongly Disagree (5)]

The analyst DID NOT create rapport with the customer

The analyst tried to influence the customer

The analyst interrupted the customer

The analyst DID NOT create rapport with the customer

The analyst tried to influence the customer

The analyst interrupted the customer

Please rate your agreement with the following statements about TEAMWORK and PLANNING * [Strongly Agree (1), Agree (2), Not sure (3), Disagree (4), Strongly Disagree (5)]

There was lack of coordination and choreography among team members

The analyst did NOT manage their time in a proper way

The analyst showed a lack of preparation on the domain

The analyst looked like they did not plan the interview

There were long pauses during the interview

There was lack of coordination and choreography among team members

The analyst did NOT manage their time in a proper way

The analyst showed a lack of preparation on the domain

The analyst looked like they did not plan the interview

There were long pauses during the interview

Appendix 2: Group performance based on SRS Document Assessment

We had three groups each for top marks, average marks and the lowest marks. We were interested to see whether their performance during the interviews had any correlation with their understanding that leads to writing the SRS document. For ease of visualization, we have divided the interview themes into further two categories, i.e. domain-specific aspects of elicitation interview (question formulation, question omission and interview order) and social aspect of interview (communication skills, analyst behaviour, customer interaction, and teamwork and planning). The higher scores show better performance and the lower scores show poor performance (Figs. 20, 21, 22, 23, 24, 25).

Fig. 20
figure 20

Performance of Groups with top marks in domain-specific aspects of interview

Full size image
Fig. 21
figure 21

Performance of Groups with top marks in social aspects of interview

Full size image
Fig. 22
figure 22

Performance of Groups with average marks in domain-specific aspects of interview

Full size image
Fig. 23
figure 23

Performance of Groups with average marks in social aspects of interview

Full size image
Fig. 24
figure 24

Performance of Groups with lowest marks in domain-specific aspects of interview

Full size image
Fig. 25
figure 25

Performance of Groups with lowest marks in social aspects of interview

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bano, M., Zowghi, D., Ferrari, A. et al. Teaching requirements elicitation interviews: an empirical study of learning from mistakes. Requirements Eng 24, 259–289 (2019). https://doi.org/10.1007/s00766-019-00313-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00766-019-00313-0

Keywords

Search

Navigation