Development of a valid and reliable video-based decision-making test for Australian football umpires

doi:10.1016/j.jsams.2013.08.001

Journal of Science and Medicine in Sport

Volume 17, Issue 5, September 2014, Pages 552-555

https://doi.org/10.1016/j.jsams.2013.08.001 Get rights and content

Abstract

Objectives

To develop a valid and reliable video-based decision-making test to examine and monitor the decision-making performance of Australian football umpires.

Design

Validation assessments with test re-test reliability.

Methods

A video-based decision-making test was developed from a pool of 156 video-based decision-making situations. Australian football umpires (n = 56) and players (n = 45) participated in two separate phases of analysis. In phase one, players completed a test re-test reliability assessment with a 100 video-clips. Results indicated 24 clips were a reliable measure of decision-making performance. In phase two, umpires completed a test re-test protocol with 80 clips, 24 of which were the reliable clips identified by the player cohort in phase one. These 24 clips provided a measure of construct validity. Face and content validity were assessed by skill acquisition specialists, expert umpire coaches, and umpires.

Results

From each of phase one and two of the reliability assessment, 24 clips were found to have a kappa value greater than 0.30, providing a total of 48 reliable video-clips. Construct validity was supported, with the umpire group performing significantly better than the player group on the 24 clips presented to both groups on each testing occasion. Face and content validity were also demonstrated.

Conclusions

This investigation demonstrated the ability prospectively to determine reliability and validity of the video-based decision-making test designed specifically for Australian football umpires. Establishing the validity and reliability of the video clips ensures future investigations can accurately and consistently measure the decision-making performance of Australian football umpires.

Introduction

Effective decision-making is a key component of in-game performance for both athletes¹ and umpires,2, 3 with skilled performance characterised by consistent, efficient, correct decision-making throughout a game.¹ Numerous studies have developed methods to assess the decision-making skill of soccer4, 5 and rugby officials,⁶ however, only one study⁷ to date has examined how to assess and/or monitor the decision-making performance of Australian football (AF) umpires.

Investigations have demonstrated the potential for sport-specific video-based decision-making tests to differentiate performance of skilled and less skilled officials.2, 7 Video-based tests are seen as an appropriate method to assess decision-making performance because of the ability to present developing, in-game decision-making situations, which are probably more ecologically valid than presenting static images and provide greater experimental control than in-game testing. Currently, there are no standardised decision-making assessment tools for AF umpires. Therefore, from a practical perspective, a valid and reliable video-based decision-making test may be used to identify future umpire talent, monitor decision-making development, and establish skill benchmarks for umpire selection purposes.

Despite encouraging findings for the use of video-based methods to assess the decision-making skills of sports officials,2, 5, 7 a limitation is that no umpire-based study has reported the reliability and/or validity of the video-based test. Without this assessment the test may not measure what it claims to measure, and it is not possible to determine the consistency of responses obtained.⁸ This is problematic because it is unclear whether differences in umpire decision-making performance are a result of skill differences or an unreliable test. Therefore, further empirical research is needed to determine the validity and reliability of video-based tests before the continued use of such measures.

New video-based decision-making tests should be based on the recognised theoretical frameworks that underpin sport-based decision-making performance. Consequently, video information must provide a decision-making situation that enables the generation of possible outcomes before selecting an appropriate decision.⁹ For a test to be specific to an umpire cohort, the video should present scenarios where participants can apply their knowledge of game rules.¹⁰ By acknowledging the theoretical underpinnings of sport-based decision-making, it is likely that the quality of decision-making tests will be improved. Therefore, the aims of the current study were to develop a video-based test to examine the decision-making performance of AF umpires and establish the validity and reliability of this test to assess decision-making performance.

Section snippets

Method

AF umpires (n = 56) were recruited from Division 1 competitions (i.e., local level) in metropolitan and regional Victoria (Australia). An additional group of AF players (n = 45) from regional Division 1 and 2 competitions in Victoria were selected to enable construct validity to be established and provide an appropriate sample size for reliability tests. All players were limited to no experience in umpiring competitive AF games. Ethics approval was granted by the University's Human Research Ethics

Results

The preliminary set of 100 video-clips (i.e., two-50 clip tests) presented to the player cohort were analysed for test re-test reliability (phase one). Results indicated that 24 clips had a kappa value greater than 0.30, which is congruent with Landis and Koch's¹⁸ definition of fair agreement. Analysis of the 56 new clips presented to the umpire cohort (phase two) identified a further 24 clips that had a kappa value greater than 0.30. Accordingly, a total of 48 clips were considered reliable

Discussion

This study is the first to develop and assess the validity and reliability of an AF umpire specific video-based decision-making test. The results present a novel method to prospectively determine the validity and reliability of a video-based decision-making test, which has not been previously reported in the umpire research literature. Whilst video-based methods of assessment have been commonly used to evaluate decision-making performance of sporting officials,4, 6, 7 these studies have failed

Conclusion

In conclusion, the results of this study demonstrate that new video-based decision-making tests can be prospectively assessed for validity and reliability. Specifically, this study provided evidence that 48 video-clips can consistently measure AF umpires’ decision-making skill. Therefore, future investigations can use this test as a valid and reliable measure to assess or monitor decision-making performance changes during training interventions. Furthermore, this study indicates the stages and

Practical implications

•
The video-based test can be used to reliably guide on-going assessment and monitoring of decision-making performance of AF umpires.
•
The new video-based test has the potential to be used in determining skill benchmarks and talent identification processes for AF umpires.
•
Investigations could utilise these protocol methods to develop training programmes that develop decision-making skill within AF umpires.

Acknowledgement

The authors sincerely thank the Australian Football League Research Board for funding this project.

References (21)

J.G. Johnson
Cognitive modelling of decision making in sports
Psychol Sport Exerc
(2006)
J.F. Gréhaigne et al.
The teaching and learning of decision making in team sports
Quest
(2001)
P. Catteeuw et al.
Decision-making skills, role specificity, and deliberate practice in association football refereeing
J Sports Sci
(2009)
C. MacMahon et al.
Referee decision making in a video-based infraction detection task: application and training considerations
Int J Sports Sci Coach
(2007)
P. Catteeuw et al.
Perceptual-cognitive skills in offside decision making: expertise and training effects
J Sport Exerc Psychol
(2010)
G. Schweizer et al.
A video-based training method for improving soccer referees’ intuitive decision-making skills
J Appl Sport Psychol
(2011)
D.R. Mascarenhas et al.
Training accurate and coherent decision making in rugby union referees
Sport Psychol
(2005)
P. Larkin et al.
Perceptual and decision making skills of Australian football umpires
Int J Perf Anal Sport
(2011)
W.G. Hopkins
Measures of reliability in sports medicine and science
Sports Med
(2000)
D. Farrow et al.
A recipe for expert decision making Chapter 10

There are more references available in the full text version of this article.

Cited by (20)

Using 360° virtual reality as a decision-making assessment tool in sport
2019, Journal of Science and Medicine in Sport
Citation Excerpt :
There are only a finite amount of games with inherent physical loads, which has led to the development of additional methods to investigate decision-making. To investigate and assess decision-making, video-based methods are commonly used as they provide a controlled off-field environment for testing and training both athletes8 and officials.9 Due to the high variability of match decision-making, video-based assessments provide a reliable and valid measure of decision-making performance.
To examine the reliability, construct validity and ecological validity of 360° VR and match broadcast footage for off-field decision-making assessment in Australian football umpires.
Validation assessments with test re-test reliability.
Two video-based tests of 60 clips each were developed to assess Australian football umpire decision-making, including 360° video of small-sided Australian football games and match broadcast footage of AFL games. Elite (n = 13) and amateur (n = 15) umpires participated in two testing sessions, in a randomised, counterbalanced design. Test re-test reliability was assessed using Cohen’s Kappa for individual clips and Intraclass Correlation Coefficients for test scores. Video tests were assessed for construct validity. Ecological validity of the decision-making processes was assessed for each method.
31 clips met the minimum Kappa criteria for the 360° VR test and 28 clips for match broadcast. Results indicated strong reliability for the 360° VR (ICC = 0.89) and match broadcast (ICC = 0.89) tests. For both video modes, elite umpires performed significantly better in decision-making accuracy than amateur (p < 0.05). For ecological validity of the decision-making processes, 360° VR was rated significantly higher than match broadcast vision (p < 0.05) overall.
This is the first study to examine the reliability and validity of 360° VR footage as an off-field decision-making assessment tool in sport. As match broadcast vision is commonly used to assess decision-making in athletes and officials, results suggest that 360° VR is also an appropriate assessment tool. Although both video modes demonstrate similar reliability and construct validity, 360° VR was considered more specific to in-game decision-making processes, suggesting stronger ecological validity.
Video-based testing in sporting officials: A systematic review
2019, Psychology of Sport and Exercise
Citation Excerpt :
They concluded sport-specific motor experience may play an important role in decision-making skill. Researchers are encouraged to assess accuracy of decisions, as determined by a panel of experts such as officials’ coaches (Larkin, Mesagno, et al., 2014; Spitz et al., 2018). Rugby union referees have been compared to touch-judges, assessors, and referee coaches (Mascarenhas et al., 2005a).
Decision-making is the most important skill for sporting officials, consequently, assessment of this skill is becoming increasingly popular in the literature. There is considerable interest in the use of video-based methods to assess decision-making of officials in controlled, off-field environments.
Systematic review of the literature examining video-based testing in sporting officials.
Using the keywords “umpire”, “referee”, “sport officials”, “decision making” and “judgement”, a comprehensive search was conducted in February 2018 on electronic databases (SPORTDiscus, Medline, PsycInfo, Google Scholar). Inclusion criteria included full text articles from January 2000 to January 2018 published in peer-reviewed journals. Only ‘central’ or ‘field’ officials were included in this review (i.e., assistant referees, touch judges were excluded).
The search yielded 27 studies. The majority of articles were specific to soccer officials. Overall, video-based testing appears to be a valid measure of decision-making differentiating between performance levels. This review highlighted a high degree of variability among the methods applied, with varied participation groups, clip type used, and influences on decision-making. The reporting of reliability and implementation of transfer tests was rarely incorporated in the research.
Video-based testing appears to be a valid measure of decision-making of officials in an off-field, controlled environment. This research area would be advanced through further investigation into sports other than soccer, examination of transfer to match performance testing, reporting the reliability of the test, reporting decisional accuracy rather than solely number of decisions, and investigation of additional video modes.
The influence of match characteristics and experience on decision-making performance in AFL umpires
2019, Journal of Science and Medicine in Sport
The purpose of this study was to investigate the effects of match characteristics and elite level umpiring experience on the decision-making performance of umpires in the Australian Football League (AFL).
Cross Sectional Study.
All decisions from the 2014 AFL home and away season made by 32 male AFL field umpires (age 33.1 ± 5.0 years, AFL umpire experience 122 ± 105 matches) were classified by a panel of four expert umpire coaches into correct, missed and unwarranted categories. The 8001 decisions were then further categorised based on the match time, score differential, field location, home ground advantage, match attendance and elite level umpiring experience. Chi-square with standardised residuals and two-way analysis of variance were used to analyse the data.
Higher elite level umpiring experience and a larger score differential at the time of the decision resulted in improved decision-making accuracy. Home ground advantage, attendance and match time did not influence the decision-making performance of umpires. Impact averse decisions occurred in the back line of play, with umpires tending to be conservative with decisions that may impact the score.
Decisions that have an increased influence on an umpire’s decision-making performance include score differential and elite umpiring experience. Implications on umpiring performance regarding the venue, attendance and the duration of a match were not apparent.
Faster, slower or real time? Perceptual-cognitive skills training with variable video speeds
2016, Psychology of Sport and Exercise
Citation Excerpt :
Yet, athletes are not unique in their search to acquire key performance characteristics. Nowadays, researchers and practitioners apply innovative training methods and representative tasks to ameliorate the decision-making processes of (assistant) referees, both on- and off-field (e.g., Catteeuw, Gilis, Jaspers, Wagemans, & Helsen, 2010; Larkin, Berry, Dawson, & Lay, 2011; Larkin, Mesagno, Berry, & Spittle, 2013; MacMahon, Starkes, & Deakin, 2007; Mascarenhas, Collins, Mortimer, & Morris, 2005; Put et al., 2014). Using Hogarth (2008) learning approach to intuition,1 Schweizer, Plessner, Kahlert, and Brand (2011), for example, demonstrated that the integration of online video sequences with foul/no-foul situations can improve the decisions of association football referees, particularly when immediate feedback on the correctness of decisions was provided.
We examined whether a short-term, off-field training intervention using different video speed manipulations (i.e., normal speed, 75%, and 125% of normal speed), enhances the offside decision-making performance of international level assistant referees. Moreover, we explored the optimal sequence of various video speed manipulations during training.
All groups were exposed to a pre- and posttest. During acquisition, the assistant referees assessed 60 offside video simulations during three different training sessions. The order of the video speed conditions varied during the training intervention depending on participant group assignment: increasing speed group (n = 33; 75%–100%–125%); decreasing speed group (n = 33; 125%–100%–75%); and arbitrary speed group (n = 30; 100%–75%–125%).
First, the decision-making accuracy was higher during the presentation of real time and faster video conditions when compared to the slower video condition. Second, only the decreasing speed group improved their offside decision-making performance from pre- to posttest.
We conclude that scheduling a decreasing video speed sequence throughout training is more beneficial to enhance learning than increasing or arbitrary speed conditions. Although it is too premature to replace existing training protocols that typically use constant speed videos, the present study offers evidence to suggest that training interventions for elite performers that decreasing the speed of exposure during the learning process may have some benefits over regular speed presentations.
Application of Simulation Technology in Football Training: A Systematic Review of Empirical Studies
2024, Open Sports Sciences Journal
Talent Identification and Development in Sport
2023, Sport and Exercise Psychology: Theory and Application

View all citing articles on Scopus

View full text

Original researchDevelopment of a valid and reliable video-based decision-making test for Australian football umpires

Abstract

Objectives

Design

Methods

Results

Conclusions

Introduction

Section snippets

Method

Results

Discussion

Conclusion

Practical implications

Acknowledgement

Psychol Sport Exerc

The teaching and learning of decision making in team sports

Quest

Decision-making skills, role specificity, and deliberate practice in association football refereeing

J Sports Sci

Referee decision making in a video-based infraction detection task: application and training considerations

Int J Sports Sci Coach

Perceptual-cognitive skills in offside decision making: expertise and training effects

J Sport Exerc Psychol

A video-based training method for improving soccer referees’ intuitive decision-making skills

J Appl Sport Psychol

Training accurate and coherent decision making in rugby union referees

Sport Psychol

Perceptual and decision making skills of Australian football umpires

Int J Perf Anal Sport

Measures of reliability in sports medicine and science

Sports Med

A recipe for expert decision making Chapter 10

Original research
Development of a valid and reliable video-based decision-making test for Australian football umpires