Incorporating deep uncertainty into the elementary effects method for robust global sensitivity analysis
Introduction
To provide a scientific basis for decision-making, social–ecological modelling is used more than ever to provide future projections of natural resource use, economic activities, environmental impacts, and their interplay at local to global scales (Wise et al., 2009, Nelson et al., 2010, Bateman et al., 2013, Liu et al., 2015). However, modelling and model diagnostics techniques, such as global sensitivity analyses (GSA), are challenged by the presence of deep uncertainty, where probabilities of the occurrence of future events are unknown, the uncertainty is uncontrollable, and predictions based on past data are unreliable (Knight, 1921, Wintle et al., 2010, Cox, 2012, McInerney et al., 2012). A common and effective way of coping with deep uncertainty is to characterise a range of scenarios, each of which is a structured account of a plausible future (Peterson et al., 2003, Wilkinson and Kupers, 2013, Kirby et al., 2014, Hatfield-Dodds et al., 2015, Kirby et al., 2015). Recent influential examples include the Representative Concentration Pathways (Moss et al., 2010, van Vuuren et al., 2011) and the Millennium Ecosystem Assessment (2005). A key characteristic of scenarios is internal consistency. In essence, parameter settings for each scenario are logical when taken together and varying individual parameters risks illogical or impossible parameter combinations. Scenarios should not include contradicting assumptions and must be regarded as plausible stories of the future by experts (van Vuuren et al., 2011).
In modelling complex social–ecological systems, GSA is seen as an increasingly important component which provides insights about the mapping of model inputs to model outputs, and major parametric uncertainty sources (Saltelli et al., 2000). It enables the quantification of the influence of uncertainty in model input parameters on the variability of model outputs (Saltelli et al., 2008). Information provided by GSA enables modellers to verify models and identify errors, to understand the structure of complex models, to prioritize influential parameters for data collection and refinement to improve the model accuracy and reduce uncertainty, and to identify benign parameters which can be safely ignored. GSA has been applied to a wide range of ecological and environmental models including hydrology (e.g., Nossent et al., 2011, Shin et al., 2013, Gan et al., 2014, Peeters et al., 2014), land use (e.g., Gao et al., 2015), forestry (e.g., Song et al., 2012, Song et al., 2013), agriculture (e.g., DeJonge et al., 2012, Zhao et al., 2014), and population dynamics (e.g., Fieberg and Jenkins, 2005). Several methods for GSA have been proposed including screening (e.g., Morris, 1991), non-parametric (e.g., Saltelli and Marivoet, 1990), variance-based (e.g., Sobol’, 1993, Saltelli et al., 2010), density-based (e.g., Liu and Homma, 2009), and expected-value-of-information-based methods (e.g., Oakley et al., 2010). Two key outputs from GSA are the main or first-order effects (variance contribution of an individual parameter to the total model variance) and the total effects (variance contribution resulting from the first-order effect of an individual parameter and all its interactions with other parameters). The extended Fourier Amplitude Sensitivity Test (eFAST) method (Saltelli et al., 1999) is a state-of-the-art GSA approach which can efficiently calculate both the main and total effects (Zhao et al., 2014). However, eFAST remains relatively computationally-intensive compared to the elementary effects (EE) method (Morris, 1991) which can approximate the total effects with greatly reduced computational demand (Campolongo et al., 2007, Herman et al., 2013). It has proven to be among the most efficient parameter screening methods and has been widely applied (e.g., Song et al., 2012, Herman et al., 2013, Zhan et al., 2013).
In broad terms, GSA works by varying individual model inputs within a specified range of uncertainty in a structured way, running the model under each parameter combination, and quantifying the sensitivity of outputs to variation in inputs. However, under deep uncertainty, varying the attributes of scenarios independently may invalidate their internal consistency, risking implausible or impossible parameter combinations. Gao et al. (2015) found statistically significant differences between scenarios in the influential and non-influential parameters identified, and in parameter influence, both in terms of their ranking and in the magnitude of their total effects. Gao et al. (2015) proposed a robust global sensitivity analysis (RGSA) approach by employing four decision criteria in determining a set of sensitivity indicators that are robust to deeply uncertain futures represented as scenarios. Each criterion was used to calculate a robust sensitivity indicator based on the sensitivity indices from the eFAST method under different scenarios. However, the computational load of the eFAST method limited the utility of eFAST as a robust GSA as Gao et al. (2015) had to run the model at a coarse resolution, even using high-performance computing and parallel programming techniques (Bryan, 2013). For analysing the sensitivity of large models there is a need to incorporate robustness into a more computationally-efficient GSA method such as the EE method.
In this paper, we modified the EE method to perform a robust elementary effects (rEE) global sensitivity analysis of the Australian continental Land Use Trade-Offs (LUTO) model (Bryan et al., 2014, Connor et al., 2015) across four global scenarios. The EE method overcomes the limitations of derivative-based methods (Saltelli et al., 2008) and measures global sensitivity by sampling throughout p-level parameter space (p is the number of levels to which each dimension of the parameter space is divided). We incorporated internally-consistent scenarios into the p-level sampling space and evaluated its capability for RGSA. The effectiveness of the rEE method was assessed by statistically comparing its robust sensitivity effects with the estimates of Gao et al. (2015) obtained by applying four decision criteria based on their eFAST-calculated measures of first-order and total effects. We discuss the advantages and limitations of rEE as a method for performing global sensitivity analysis that is robust to deep uncertainty.
Section snippets
Brief overview of model and scenarios
The LUTO model provides a comprehensive assessment of Australia's future land use and ecosystem services (e.g. food, carbon, energy, biodiversity, and water) under external drivers of global change and domestic policy. The details of the LUTO model are presented by Bryan et al. (2014) and Connor et al. (2015) and are summarised here. The LUTO model estimates the potential extent and impacts of land use change at a spatial resolution of ∼1.1 km grid cells across the 85.3 million hectares of
Robust sensitivity analysis methods
The four external scenario inputs were incorporated into the rEE method to provide a robust global sensitivity analysis, giving a comprehensive assessment of the influence of each input parameter on the variability of model outputs. Then, the result was compared with robust sensitivity effects produced by applying the four decision criteria from decision theory (maximax, weighted average, minimax regret, and limited degree of confidence) to eFAST sensitivity indices (Gao et al., 2015).
Uncertainty of model outputs
In the rEE experiments, variation in the 24 output variables was captured from the 510 runs. Empirical frequency distributions calculated via kernel density estimation of these outputs (overlaid on the box plots) are shown in Fig. 1. The shade shape of a violin plot represents the probability density of the output. Some outputs such as ProductionDairy and ProductionSheep had a more normal distribution, while the distributions of others such as AreaWheat(Biofuel) and AreaWoodyPerennials(Biofuel)
Discussion
We have evaluated the ability of the elementary effects method to perform global sensitivity analysis that is robust to deep uncertainty characteristic of future scenarios. Overall, we observed moderate to strong correlations between the robust sensitivity indicator produced by the rEE method and the eight RGSA indicators derived from the four decision criteria applied to the eFAST first-order and total sensitivity indices. The ranking of individual model input parameters in order of rEE
Conclusion
We investigated the potential of the EE method as an approach to global sensitivity analysis that is robust to deep uncertainty. By incorporating scenario inputs into the method and comparing the results with those obtained from an eFAST-based RGSA method, we found the rEE method did not fully deliver what was expected. When models are complex and behave in a strongly non-linear way, the rEE method did not provide an accurate sensitivity quantification under deeply uncertain conditions.
Acknowledgements
The work is supported by CSIRO Agriculture and Australian National Outlook initiative.
References (56)
High-performance computing tools for the integrated assessment and modelling of social–ecological systems
Environ. Modell. Softw.
(2013)- et al.
Supply of carbon sequestration and biodiversity services from Australia's agricultural land under global change
Global Environ. Change
(2014) - et al.
Thinking inside the box: a participatory, computer-assisted approach to scenario discovery
Technol. Forecasting Soc. Change
(2010) - et al.
An effective screening design for sensitivity analysis of large models
Environ. Modell. Softw.
(2007) - et al.
Modelling Australian land use competition and ecosystem services with food price feedbacks at high spatial resolution
Environ. Modell. Softw.
(2015) - et al.
Variance-based sensitivity analysis for wastewater treatment plant modelling
Sci. Total Environ.
(2014) - et al.
Global sensitivity and uncertainty analysis of a dynamic agroecosystem model under different irrigation treatments
Ecol. Model.
(2012) - et al.
Land use mapping error introduces strongly-localised, scale-dependent uncertainty into land use and ecosystem services modelling
Ecosyst. Serv.
(2015) - et al.
Assessing uncertainty in ecological systems using global sensitivity analyses: a case example of simulated wolf reintroduction effects on elk
Ecol. Model.
(2005) - et al.
A comprehensive evaluation of various sensitivity analysis methods: a case study with a hydrological model
Environ. Model. Softw.
(2014)