Can end-users' flood management decision making be improved by information about forecast uncertainty?

Abstract

In the course of the D-PHASE project, a visualisation platform was created, which provided a large amount of meteorological and hydrological information that was used not only by scientists, but also by scientifically aware laypeople in the field of flood prevention. This paper investigates the benefits of the platform for its end-users' situation analysis and decision making, and in particular, its usefulness in providing an ensemble of models instead of already interpreted forecasts. To evaluate the platform's impact on users in Switzerland, a panel approach was used. Twenty-four semi-standardized questionnaires were completed at the beginning of the demonstration phase and 27 questionnaires were completed five months later. The results suggest that the platform was perceived as adding value to both situation analysis and decision making, and helped users to feel more confident about both. Interestingly, users' preference for receiving complex, primary information and forming their own impressions over receiving interpreted information and recommendations increased during the demonstration phase. However, no actual improvement in the quality of decisions was reported.

Introduction

How useful or beneficial is uncertainty information in meteorological and hydrological forecasting? Discussion of this issue has recently been intense; especially since ensemble prediction systems have become popular and widely used (see Rossa et al., 2011-this issue). Increased uncertainty affects the users of forecast information in different ways. First, they have to get used to new forms of data presentation in ensemble predictions; second, they have to learn to interpret the ensemble systems; and third, they have to find ways to incorporate the additional information in their procedures. The challenges will be greatest when people with only a basic knowledge of meteorology or hydrology are exposed to end-to-end forecasting systems such as the one developed during the D-PHASE project, i.e. users who are further “down” the alert chain. Thus, such forecasting systems have to be designed to meet the needs of a wide variety of users. In this sense, there is a danger that “advances in forecast technological sophistication are moving well ahead of end-users' abilities to take advantage of them” (McCarthy, 2007). The research presented here aims to describe how such a forecasting system has been adopted and to what extent the new information affects the users' decision processes.

D-PHASE (Demonstration of Probabilistic Hydrological and Atmospheric Simulation of flood Events in the Alpine region, Rotach et al. 2009) is a World Weather Research Project (WWRP) Forecast Demonstration Project (FDP). It aims to demonstrate some of the achievements of the Mesoscale Alpine Programme (Bougeault et al. 2001); in particular, its ability to forecast heavy precipitation and related flooding events in the Alpine region. To this end, a distributed, real-time, end-to-end forecasting system for heavy precipitation and subsequent flood events in the Alpine region has been established. The key element of this forecasting system is a web-based visualisation platform. The platform features an innovative graph-based presentation of the forecasts from an ensemble of 30 meteorological models for most areas and of several subsequent hydrological models for each river runoff gauging station. The most widely used meteorological models by Swiss users were COSMOCH2, COSMOCH7, COSMO-LEPS, and PEPS, the most popular hydrological models were PREVAH based on COSMO-LEPS/COSMOCH2/COSMOCH7 and HBV based on COSMO-LEPS, PEPS, COSMOCH2, or COSMOCH7 (for more details about these models see Rotach et al., 2009, Zappa et al., 2008). The visualisation platform is entered through a map-based alert overview over the whole D-PHASE region, which is the Alpine arch and adjacent regions. The colour codes (indicating events with return periods of 60 days, 180 days, and 10 years) indicate whether a precipitation or runoff alert level has been reached by any of the models for any of the regions. From this “level 1” overview, users can select their country region of interest (“level 2”, see picture A in Fig. 1). Next to the regional map, a table summarizes the states of all atmospheric and hydrologic models for all catchments (picture B in Fig. 1). Choosing one of the level 3 catchments (with average sizes of ca. 2000 km² in Switzerland) or one of the gauging stations (18 in Switzerland), the detailed information for the region or station can be retrieved (picture C in Fig. 1). The outputs of all models available for the catchment/station are listed here showing temporal progress of alert states and peak amounts in 3 h, 6 h, 12 h and 24 h forecast ranges, where for the probabilistic models the probability of exceedance is given for each time step on display. Further quantified uncertainty information from each of the models is available in the form of graphical products of the direct model output, such as precipitation accumulation (picture D in Fig. 1). In addition to these forecasts and alerts direct links to nowcasting products from the involved forecasting centres are available on the site. For more information about D-PHASE, see the overviews of Rotach et al., 2009, Zappa et al., 2008 or http://www.map.meteoswiss.ch/d-phase.

The visualisation platform became accessible during the D-PHASE Operations Period (DOP) in several Alpine countries in 2007, and has been maintained and optimized since the DOP. In Switzerland, about 65 users from 32 institutions were officially registered during the DOP. They consisted of hydrological and meteorological forecasters, natural hazard experts from administration offices, and people in charge of flood management and civil protection.

The D-PHASE Operations Period was evaluated on several levels, including objective verification of the forecasts (see Ament et al., 2011-this issue) and an evaluation of the users' reactions. For example, forecasters rated the prognostic value of the models used in an evaluation protocol after each shift, end users' comments were collected when they completed an online feedback form, an evaluation workshop was held, and qualitative interviews were conducted to assess the change in working procedures, in communication, and in mutual perceptions between the end users and the forecasting services. This paper focuses on a standardized evaluation of users' perceptions and knowledge.

Although humans have to make decisions under uncertain conditions almost every day, they often do so in a way, that may appear to the outside observer as being non-reasoned, intuitive, or at least unsystematic. Indeed, results from many research fields have demonstrated that users of risk or uncertainty information, ranging from the broad public to professionals, have difficulty integrating the information into a binary decision (e.g., Kahnemann and Tversky, 1979, Tversky and Kahneman, 1974). However, recent work has also shown that uncertainty information may even be preferred over deterministic information in the field of weather forecasting (Morss et al., 2008). Overall, there is still much scope for improving how forecasts are communicated. While much is known about how people perceive probabilities and frequencies, less is known about how this knowledge is used when making decisions in emergency situations (e.g., Handmer and Proudley, 2007). Different user groups of ensemble prediction systems also have varying needs (Demeritt et al., 2007) that should be investigated. In particular, we need to know more about the best ways of visualising forecasting data for end-users (Morss et al, 2008) and to introduce it in a particular communication context (“framing”, see Gigerenzer et al., 2005).

On the methodological side, existing research has been criticised for overusing controlled laboratory settings instead of “real world” settings (Morss et al., 2008). Quantitative data is available predominately for the general population and everyday weather forecasts, whereas research among professionals, and in particular among managers of natural hazards, e.g. the civil protection authorities, with regard to less frequent or extreme events is still limited and has so far relied on qualitative interviewing or focus group techniques.

One of the main goals of introducing the D-PHASE visualisation platform was to facilitate an improvement in how users analysed situations and made decisions. This goal, relating to the final outcomes of the implementation of this new instrument, had to be broken down, as the actual quality of decisions could not be assessed objectively. Thus some expected outcomes were defined, which are regarded as necessary preconditions for adequate situation analysis and decision making (see Table 1). These refer to people's understanding of forecasts and probabilistic information, and their opinions regarding the particular information, such as whether they trusted or accepted it. Distinguishing between trust, i.e. the perceived reliability of information sources or partners, and confidence, i.e. the belief that things are under control, has proven to be useful in the context of risk perception and communication (Siegrist and Cvetkovich, 2000, Siegrist et al., 2005). Therefore, an increase in the confidence of users (subjectively perceived) was regarded as desirable. All of these concepts represent the first and most important set of evaluation criteria (outcome evaluation, cf. Rossi and Freeman, 1993). To produce these outcomes, it was necessary for the information presented during the implementation of the measure (sometimes called the output of the intervention) to fulfil certain quality criteria. The quality of the information content can be assessed throughout the communication process between users and the platform, starting from: (1) exposure to the information, (2) paying attention to the material, (3) understanding it, and (4) being able to transfer it and use it in the user's own specific decision-making context (cf. Table 1). These four criteria for content quality are based on Rohrmann's (2000a) theoretical framework, which has already been applied to evaluate the content of risk communication in the field of natural hazards (e.g., Rohrmann, 2000b).

Two approaches to evaluating whether using the platform had the desired outcomes and whether the content of the information fulfilled the quality criteria are possible. First, users can be asked directly to rate the quality of the new information as well as its value or benefit shortly after using it. However, this ex-post approach is not sufficient, because the actual relevant changes that occur during the DOP regarding the expected outcomes should also be measured. We therefore formulated two different types of basic research questions, with several sub-questions that are related to the evaluation criteria of content (i.e. output) and outcome:

Research question 1: How useful and valuable do users find the new forecast information?

• 1a) Is the newly available information being used?

• 1b) Are users able to access, manage, understand, and interpret the new information?

• 1c) How do users rate the platform's benefits (e.g. the usefulness for optimizing their procedures)?

Research question 2: Does the use of D-PHASE have an impact on how users perceive and handle forecasts?

Is there a measurable change during the DOP regarding …

• 2a) … users' understanding and evaluation of available forecasts?

• 2b) … users' acceptance of uncertainty information?

• 2c) … users' self-assessment of their personal competence regarding the particular situation analysis and decision making?