Performance appraisal of software testers

https://doi.org/10.1016/j.infsof.2013.11.002Get rights and content

Highlights

  • We surveyed IT professionals about software tester appraisal practice.

  • A variety of approaches are currently used.

  • We proposed a standardized Performance Appraisal Form (PAF) for software testers.

  • We refined our proposed PAF based on feedback from project managers.

Abstract

Context

To determine the effectiveness of software testers a suitable performance appraisal approach is necessary, both for research and practice purposes. However, review of relevant literature reveals little information of how software testers are appraised in practice.

Objective

(i) To enhance our knowledge of industry practice of performance appraisal of software testers and (ii) to collect feedback from project managers on a proposed performance appraisal form for software testers.

Method

A web-based survey with questionnaire was used to collect responses. Participants were recruited using cluster and snowball sampling. 18 software development project managers participated.

Results

We found two broad trends in performance appraisal of software testers – same employee appraisal process for all employees and a specialized performance appraisal method for software testers. Detailed opinions were collected and analyzed on how performance of software testers should be appraised. Our proposed appraisal approach was generally well-received.

Conclusion

Factors such as number of bugs found after delivery and efficiency of executing test cases were considered important in appraising software testers’ performance. Our proposed approach was refined based on the feedback received.

Introduction

The reliability of delivered software, to a large extent, depends on the performance of software testers. An accurate performance appraisal of software testers is thus very important for their recruitment, monitoring and development, and for testing team performance management. Furthermore, from a research perspective, to conduct studies of factors that potentially affect software testers’ performance, a validated, reliable instrument to assess software testers’ performance is an essential prerequisite. For example, in a research study [1] investigating the influence of personality on the effectiveness of software testing the authors needed to distinguish different levels of performance. To accomplish this, a method to assess high performing software testers was necessary.

However, from an extensive search of relevant literature we did not find any widely accepted and well established performance appraisal method for software testers. Therefore, as an operational need for our research, we have proposed a new Performance Appraisal Form (PAF) for software testers. However, any such instrument should be validated for use.

In this study, we sought a broader insight into industrial practice in the area of tester performance appraisal by surveying nearly 20 software development project managers to describe the practices in their own organizations, and their own views on tester performance appraisal. We then attempted to validate the approach taken in our PAF by collecting feedback from software development project managers on the proposed PAF, obtaining detailed feedback from 10 of them. With this two-pronged approach, we not only sought direct comment on our proposed PAF, we hoped to find out whether industrial practice could further inform our PAF design, and also whether the PAF proposed could be of industrial as well as research interest.

The rest of the article is arranged as follows – Section 2 summarizes our review of relevant literature, Section 3 details our research questions, Section 4 describes the proposed Performance Appraisal Form (PAF), Section 5 describes the method of this research study, Section 6 presents our results, Section 7 lists the threats to validity of the research, Section 8 presents our discussion on the findings and finally Section 9 concludes the article.

Section snippets

Performance evaluation of software testers

As reported in our earlier research [2], there is no widely accepted instrument or approach to evaluating the performance of software testers in the academic literature. However, we found some suggestions about criteria that may be important for evaluating software testers’ performance. Fenton and Pfleeger [3] suggest measuring efficiency of software testing using the number of bugs found per KLOC. Grady and Caswell [4] suggest looking for average reported bugs per working day. However, Kaner

Research questions

Current methods utilized to appraise software testers in the software industry have not been reported or evaluated in any detail in the open literature [2]. We therefore do not know what are the relative advantages and disadvantages of current performance appraisal practices for software testers and how we can improve on these.

Based on our review of the literature and analysis of different requirements listed in job advertisements for testers, we designed a new Performance Appraisal Form (PAF)

Proposed Performance Appraisal Form (PAF)

The objective of our proposed PAF is to provide a standard assessment instrument to assess overall performance of software testers from different performance dimensions. Some performance appraisal instruments use multiple forms to assess different aspects of employee performance. However, for simplicity we chose to design an integrated form. The performance dimensions of our proposed PAF were based on different approaches [13] to performance appraisal: Performer focused appraisal: This approach

Methodology

A personal opinion survey [18] was used to conduct our survey. Compared to other available research methods, a survey enabled us to collect opinions of higher number of participants in limited available time [19]. The design of the survey was by following the steps as suggested by Kitchenham and Pfleeger [18], and these are presented in the following subsections.

Results

Our survey was divided into two main sections. We noticed that we obtained a different rate of participation for the two sections. Unfortunately, 22.2% of participants dropped out after completing the first section. Thus we describe the results obtained from each section separately.

A total of 18 participants (8 in the initial survey and 10 in the new lightweight survey) participated in this section.

Threats to validity

One of the threats that can limit the external validity of our reported results is over-generalization of the findings. We believe that the nature of participation required for this study put off many potential participants and as such we obtained a limited number of participants. Due to the small number of participants we had to modify the survey to require less time and yet still the participation rate was not satisfactory. We found that many participants were less interested in the second

Discussion

From the responses to the survey questionnaire of the state of practice of performance appraisal, it is evident that a formal process of employee appraisal is practiced in the majority of organizations. A specialized appraisal process is also common for software testers. However, when there is no such specialized process, performance of software testers is most commonly appraised using a more general employee appraisal process or a manager’s evaluation. In only a small number of organizations

Conclusion

This study aimed to obtain information about the state of practice of performance appraisal of software testers and to make suggestions on how the appraisal process can best be conducted. We found that there are two trends: some organizations use the same performance appraisal process for all employees, whereas some use a specialized one for software testers. Our participants suggested a number of criteria that should be considered in appraising performance of software testers. Among those

References (21)

  • T. Kanij, R. Merkel, J. Grundy, An empirical study of the effects of personality on software testing, in: Accepted to...
  • T. Kanij et al.

    Performance assessment metrics for software testers

  • N. Fenton et al.

    Software Metrics: A Rigorous and Practical Approach

    (1997)
  • R.B. Grady et al.

    Software Metrics: Establishing a Company-Wide Program

    (1987)
  • C. Kaner

    Don’t use bug counts to measure testers

    Softw. Test. Quality Eng.

    (1999)
  • C. Kaner, Measuring the effectiveness of software testers, in: Software Testing Analysis & Review Conference (STAR)...
  • B.L. Killingsworth et al.

    A model for motivating and measuring quality performance in information systems staff

    Inform. Syst. Manage.

    (2001)
  • D.B. Mayer et al.

    Selection and evaluation of computer personnel – the research history of sig/cpr

  • R.A. Dickmann

    A programmer appraisal instrument

  • E. Bairdain

    Research studies of programmers and programming

    IBM Corporation

    (1964)
There are more references available in the full text version of this article.

Cited by (13)

  • Recruitment, engagement and feedback in empirical software engineering studies in industrial contexts

    2018, Information and Software Technology
    Citation Excerpt :

    Piloting a study with a small number of practitioners has been demonstrated to be valuable in previous work in case study research with industry, where they recommend structuring case studies iteratively, taking early samples and pilot testing [1]. We failed to sufficiently pilot our early personality testing surveys, resulting in low engagement [14]. In contrast, we refined our worklog, tester appraisal and software AD usage studies, tasks and instruments, and GT interview questions from early feedback from such pilot studies [12,17,21].

  • Mining and Fusing Productivity Metrics with Code Quality Information at Scale

    2023, Proceedings - 2023 IEEE International Conference on Software Maintenance and Evolution, ICSME 2023
  • Performance Management of IT Professionals: A Humanistic Model

    2022, Proceedings of the 17th Conference on Computer Science and Intelligence Systems, FedCSIS 2022
  • DHuTI: A Maturity Model for Human Performance in Information Technology

    2022, Iberian Conference on Information Systems and Technologies, CISTI
View all citing articles on Scopus
View full text