Measuring nursing competencies in the operating theatre: Instrument development and psychometric analysis using Item Response Theory

Summary

Background

Concern about the process of identifying underlying competencies that contribute to effective nursing performance has been debated with a lack of consensus surrounding an approved measurement instrument for assessing clinical performance. Although a number of methodologies are noted in the development of competency-based assessment measures, these studies are not without criticism.

Research aim

The primary aim of the study was to develop and validate a Performance Based Scoring Rubric, which included both analytical and holistic scales. The aim included examining the validity and reliability of the rubric, which was designed to measure clinical competencies in the operating theatre.

Research method

The fieldwork observations of 32 nurse educators and preceptors assessing the performance of 95 instrument nurses in the operating theatre were used in the calibration of the rubric. The Rasch model, a particular model among Item Response Models, was used in the calibration of each item in the rubric in an attempt at improving the measurement properties of the scale. This is done by establishing the ‘fit’ of the data to the conditions demanded by the Rasch model.

Results

Acceptable reliability estimates, specifically a high Cronbach's alpha reliability coefficient (0.940), as well as empirical support for construct and criterion validity for the rubric were achieved. Calibration of the Performance Based Scoring Rubric using Rasch model revealed that the fit statistics for most items were acceptable.

Conclusion

The use of the Rasch model offers a number of features in developing and refining healthcare competency‐based assessments, improving confidence in measuring clinical performance. The Rasch model was shown to be useful in developing and validating a competency-based assessment for measuring the competence of the instrument nurse in the operating theatre with implications for use in other areas of nursing practice.

Introduction

The focus of the study was on developing and validating a Performance Based Scoring Rubric designed to assess the competence of the instrument nurse in the operating theatre.

The aim of preparing nurses for practice is to ensure that they are competent to manage and provide quality care for patients (Dolan, 2003, Pirie and Gray, 2007). Based on the evidence surrounding medical errors, major concerns in healthcare services include patient safety and quality of care. The competence of healthcare professionals to deliver safe, effective patient-centred care is an important factor contributing to the outcomes in patient safety (Lunney et al., 2007) with the delivery of quality care directly related to the level of competence (Meretoja and Leino-Kilpi, 2001, Winslade et al., 2007). While the purpose of education is relatively easy to define, there is a far greater challenge in trying to measure the outcome of education and ensure that competent nurses are being prepared (Edwards et al., 2001). Nurses are informed of the standard of practice expected of them with the development and acceptance of nursing competency standards; however an assessment system to support such competencies is required (Evans, 2008).

A major focus in nursing education is on the judgement of clinical performance, and, although it may appear straight forward, it is a complex process (Dolan, 2003). The challenge of assessing and measuring clinical competence is inherent in the nursing profession due to the diverse nature of nursing roles (Chambers, 1998, Dolan, 2003) including the complexity of nursing practice and human interaction (Gonczi et al., 1993), all of which are difficult to measure (Cowan et al., 2008, McGrath et al., 2006). With the transfer of nursing to the tertiary sector, there has been a review of the assessment procedures that dominated nursing education in Australia, such as the use of multiple choice tests and clinical skills tests, with a move towards competency-based assessment. The difficulty of finding an effective measure is reported in the literature as the search for a valid and reliable method of assessing competence continues (Norman et al., 2002, While, 1994). In order to be able to determine the nurse's competence in delivering the expected level of care, there is a need for the nursing profession to develop procedures that would enhance agreement about what constitutes a satisfactory performance (Dunn et al., 2000, Hager and Gillis, 1995).

Difficulties surrounding the development of valid and reliable assessment measures in nursing have resulted in the development of assessment instruments that closely measure the clinical experience, that is an increase in face validity with less focus on reliability, given that there are different variables present in the clinical setting that may influence the outcome of the assessment (McGrath et al., 2006). Although rarely addressed, the reliability and validity of assessment measures reported in the literature have not been specific enough (Watson et al., 2002) with theoretical frameworks rarely reported (Meretoja and Leino-Kilpi, 2001). Literature reviews suggest that there is no ‘gold standard’ for measuring clinical competence (Dolan, 2003, Redfern et al., 2002, Watson et al., 2002) with assessing nurses continuing to pose a challenge for nursing education (Evans, 2008).

Competence cannot be directly observed (Gonczi et al., 1993, Heywood et al., 1992, Wolf, 1989), therefore the evidence needs to be of sufficient quality and quantity to make a sound judgement about the individual's level of competence (Gonczi, 1994). With the movement in nursing from an apprenticeship model to one that focuses on the development of cognitive and caring skills (Benner, 1984), the importance of being able to measure nursing skills has resulted, since such measurement is central to the notion of professionalism and essential to nursing practice (Fotheringham, 2010, Norman et al., 1992). Similarly, evaluation of competence should include assessing competence in an integrated manner (Wolf, 1989). The method selected should be based on the relevance to what is being assessed to increase the validity of the evidence and ensure generalisability of the performance to other tasks (Bailey, 1993).

Reporting of results has been a contentious issue since the introduction of competency-based training and assessment, with much of the debate focused on reporting competence beyond the competent/not yet competent dichotomy (Gillis and Griffin, 2004). There are several scoring approaches used for evaluating performance with two main approaches, analytical and holistic scoring, identified by Anthanasou (1997). Analytical scoring requires an examination of specific aspects of the performance against a set of criteria, as opposed to judging the overall impression of the candidate's performance (Anthanasou, 1997, Perkins, 1983). Assessors may judge the entire performance as a whole but each aspect of the performance is awarded a separate score as well as the allocation of an overall score (Gillis, 2003, Goulden, 1989). An advantage of using the analytical method is the provision of more detailed information about the performance of the candidate. Strengths and weaknesses are identified allowing for tailored feedback in meeting the needs of the student.

When the overall performance is judged holistically, the method of evaluation considers the overall quality of the performance, including competent components, but does not mark them separately (Goulden, 1989). A single classification of competence level is made. A limitation of holistic scoring relates to a single score, which while useful in ranking information, provides very little diagnostic information about various aspects of the performance. If an analytical scoring process is adopted, evidence of competence needs to be evaluated against some scoring criteria, which are usually referred to as a ‘rubric’. A rubric, at its most basic, is a “scoring tool that lays out the specific expectations for an assignment” (Stevens and Levi, 2005, p. 3) which is implemented when an evaluation of the quality of performance is required. ‘Rubric’ is the technical term for allocating scores and is considered the most important aspect of the performance assessment (Gillis and Griffin, 2004, Griffin, 2000).

McGrath et al. (2006) provided a range of views of the use of generic domains of clinical competence without taking into consideration the specific skills required to practice in a specialist environment. The use of generic domains relates to concerns surrounding standardising nursing practice which is a common criticism of competency-based assessment (Evans, 2008). With the development of specific competencies, for example competencies specific to specialised areas of nursing, the assumption is that generic competencies will not be sufficient because the roles in different contexts of nursing vary considerably. Therefore the need to develop practice specific competencies has been identified (Sutton and Arbon, 1994). A lack of existing instruments dealing with the measurement of perioperative competencies was identified in the literature (Nicholson, 2005) which was supported by a report published with the revised Australian College of Operating Room Nurses Competency Standards (2006).

The aim of the study was to examine the validity and reliability of a Performance Based Scoring Rubric designed to assess the competence of the instrument nurse in the operating theatre using the Rasch model (1960). The investigation into developing a measurement instrument for determining clinical competence in nursing was driven by two main rationales. The first related to a lack of a ‘gold standard’ for judging clinical performance in nursing practice (Dolan, 2003, Redfern et al., 2002, Watson et al., 2002). The second was linked to exploring the use of Rasch model theory in validating a competency-based assessment for the operating theatre to attain empirical support for construct and content validity and identify measurement problems (Beck and Gable, 2001, Fox, 1999, Smith, 2004).