Elsevier

Ecological Modelling

Volume 321, 10 February 2016, Pages 1-9
Ecological Modelling

Incorporating deep uncertainty into the elementary effects method for robust global sensitivity analysis

https://doi.org/10.1016/j.ecolmodel.2015.10.016Get rights and content

Highlights

  • We modified the elementary effects method (rEE) for robust sensitivity analysis.

  • Deep uncertainty was incorporated to produce robust sensitivity indicators.

  • The rEE method provided an effective parameter screening under deep uncertainty.

  • The rEE method was computationally-efficient and suitable for large models.

  • However, rEE did not accurately quantify the sensitivity of complex models.

Abstract

Internally-consistent scenarios are increasingly used in social–ecological systems modelling to explore how a complex system might be influenced by deeply uncertain future conditions such as climate, population, and demand and supply of resources and energy. The presence of deep uncertainty requires model diagnostic techniques such as global sensitivity analysis to provide reliable diagnostic insights that are robust to highly uncertain future conditions. We extended the elementary effects method of Morris, which is widely used to screen important model input factors at low computational cost, by incorporating deep uncertainty via the use of scenarios, and evaluated its potential as a robust global sensitivity analysis approach. We applied this robust elementary effects (rEE) method to the highly-parameterised Australian continental Land Use Trade-Offs (LUTO) model—a complex, non-linear model with strong interactions between parameters. We compared rEE sensitivity indicators with robust global sensitivity analysis (RGSA) indicators based on the variance-based eFAST method that imposes relatively high computational demand. We found that the rEE method provided a good approximation of the main effects and was effective in screening the most influential model parameters under deep uncertainty at a greatly reduced computational cost. However, the rEE method was not able to match the accuracy of the eFAST-based method in identifying the most influential parameters in the complex LUTO model based on their total effects. We conclude that the rEE method is well-suited for screening complex models, and possibly for efficient RGSA of models with weak interaction effects, but not for RGSA of complex models. Despite its limitations, rEE is a valuable addition to the robust global sensitivity analysis toolbox, helping to provide insights into model performance under deep uncertainty.

Introduction

To provide a scientific basis for decision-making, social–ecological modelling is used more than ever to provide future projections of natural resource use, economic activities, environmental impacts, and their interplay at local to global scales (Wise et al., 2009, Nelson et al., 2010, Bateman et al., 2013, Liu et al., 2015). However, modelling and model diagnostics techniques, such as global sensitivity analyses (GSA), are challenged by the presence of deep uncertainty, where probabilities of the occurrence of future events are unknown, the uncertainty is uncontrollable, and predictions based on past data are unreliable (Knight, 1921, Wintle et al., 2010, Cox, 2012, McInerney et al., 2012). A common and effective way of coping with deep uncertainty is to characterise a range of scenarios, each of which is a structured account of a plausible future (Peterson et al., 2003, Wilkinson and Kupers, 2013, Kirby et al., 2014, Hatfield-Dodds et al., 2015, Kirby et al., 2015). Recent influential examples include the Representative Concentration Pathways (Moss et al., 2010, van Vuuren et al., 2011) and the Millennium Ecosystem Assessment (2005). A key characteristic of scenarios is internal consistency. In essence, parameter settings for each scenario are logical when taken together and varying individual parameters risks illogical or impossible parameter combinations. Scenarios should not include contradicting assumptions and must be regarded as plausible stories of the future by experts (van Vuuren et al., 2011).

In modelling complex social–ecological systems, GSA is seen as an increasingly important component which provides insights about the mapping of model inputs to model outputs, and major parametric uncertainty sources (Saltelli et al., 2000). It enables the quantification of the influence of uncertainty in model input parameters on the variability of model outputs (Saltelli et al., 2008). Information provided by GSA enables modellers to verify models and identify errors, to understand the structure of complex models, to prioritize influential parameters for data collection and refinement to improve the model accuracy and reduce uncertainty, and to identify benign parameters which can be safely ignored. GSA has been applied to a wide range of ecological and environmental models including hydrology (e.g., Nossent et al., 2011, Shin et al., 2013, Gan et al., 2014, Peeters et al., 2014), land use (e.g., Gao et al., 2015), forestry (e.g., Song et al., 2012, Song et al., 2013), agriculture (e.g., DeJonge et al., 2012, Zhao et al., 2014), and population dynamics (e.g., Fieberg and Jenkins, 2005). Several methods for GSA have been proposed including screening (e.g., Morris, 1991), non-parametric (e.g., Saltelli and Marivoet, 1990), variance-based (e.g., Sobol’, 1993, Saltelli et al., 2010), density-based (e.g., Liu and Homma, 2009), and expected-value-of-information-based methods (e.g., Oakley et al., 2010). Two key outputs from GSA are the main or first-order effects (variance contribution of an individual parameter to the total model variance) and the total effects (variance contribution resulting from the first-order effect of an individual parameter and all its interactions with other parameters). The extended Fourier Amplitude Sensitivity Test (eFAST) method (Saltelli et al., 1999) is a state-of-the-art GSA approach which can efficiently calculate both the main and total effects (Zhao et al., 2014). However, eFAST remains relatively computationally-intensive compared to the elementary effects (EE) method (Morris, 1991) which can approximate the total effects with greatly reduced computational demand (Campolongo et al., 2007, Herman et al., 2013). It has proven to be among the most efficient parameter screening methods and has been widely applied (e.g., Song et al., 2012, Herman et al., 2013, Zhan et al., 2013).

In broad terms, GSA works by varying individual model inputs within a specified range of uncertainty in a structured way, running the model under each parameter combination, and quantifying the sensitivity of outputs to variation in inputs. However, under deep uncertainty, varying the attributes of scenarios independently may invalidate their internal consistency, risking implausible or impossible parameter combinations. Gao et al. (2015) found statistically significant differences between scenarios in the influential and non-influential parameters identified, and in parameter influence, both in terms of their ranking and in the magnitude of their total effects. Gao et al. (2015) proposed a robust global sensitivity analysis (RGSA) approach by employing four decision criteria in determining a set of sensitivity indicators that are robust to deeply uncertain futures represented as scenarios. Each criterion was used to calculate a robust sensitivity indicator based on the sensitivity indices from the eFAST method under different scenarios. However, the computational load of the eFAST method limited the utility of eFAST as a robust GSA as Gao et al. (2015) had to run the model at a coarse resolution, even using high-performance computing and parallel programming techniques (Bryan, 2013). For analysing the sensitivity of large models there is a need to incorporate robustness into a more computationally-efficient GSA method such as the EE method.

In this paper, we modified the EE method to perform a robust elementary effects (rEE) global sensitivity analysis of the Australian continental Land Use Trade-Offs (LUTO) model (Bryan et al., 2014, Connor et al., 2015) across four global scenarios. The EE method overcomes the limitations of derivative-based methods (Saltelli et al., 2008) and measures global sensitivity by sampling throughout p-level parameter space (p is the number of levels to which each dimension of the parameter space is divided). We incorporated internally-consistent scenarios into the p-level sampling space and evaluated its capability for RGSA. The effectiveness of the rEE method was assessed by statistically comparing its robust sensitivity effects with the estimates of Gao et al. (2015) obtained by applying four decision criteria based on their eFAST-calculated measures of first-order and total effects. We discuss the advantages and limitations of rEE as a method for performing global sensitivity analysis that is robust to deep uncertainty.

Section snippets

Brief overview of model and scenarios

The LUTO model provides a comprehensive assessment of Australia's future land use and ecosystem services (e.g. food, carbon, energy, biodiversity, and water) under external drivers of global change and domestic policy. The details of the LUTO model are presented by Bryan et al. (2014) and Connor et al. (2015) and are summarised here. The LUTO model estimates the potential extent and impacts of land use change at a spatial resolution of ∼1.1 km grid cells across the 85.3 million hectares of

Robust sensitivity analysis methods

The four external scenario inputs were incorporated into the rEE method to provide a robust global sensitivity analysis, giving a comprehensive assessment of the influence of each input parameter on the variability of model outputs. Then, the result was compared with robust sensitivity effects produced by applying the four decision criteria from decision theory (maximax, weighted average, minimax regret, and limited degree of confidence) to eFAST sensitivity indices (Gao et al., 2015).

Uncertainty of model outputs

In the rEE experiments, variation in the 24 output variables was captured from the 510 runs. Empirical frequency distributions calculated via kernel density estimation of these outputs (overlaid on the box plots) are shown in Fig. 1. The shade shape of a violin plot represents the probability density of the output. Some outputs such as ProductionDairy and ProductionSheep had a more normal distribution, while the distributions of others such as AreaWheat(Biofuel) and AreaWoodyPerennials(Biofuel)

Discussion

We have evaluated the ability of the elementary effects method to perform global sensitivity analysis that is robust to deep uncertainty characteristic of future scenarios. Overall, we observed moderate to strong correlations between the robust sensitivity indicator produced by the rEE method and the eight RGSA indicators derived from the four decision criteria applied to the eFAST first-order and total sensitivity indices. The ranking of individual model input parameters in order of rEE

Conclusion

We investigated the potential of the EE method as an approach to global sensitivity analysis that is robust to deep uncertainty. By incorporating scenario inputs into the method and comparing the results with those obtained from an eFAST-based RGSA method, we found the rEE method did not fully deliver what was expected. When models are complex and behave in a strongly non-linear way, the rEE method did not provide an accurate sensitivity quantification under deeply uncertain conditions.

Acknowledgements

The work is supported by CSIRO Agriculture and Australian National Outlook initiative.

References (56)

  • L. Gao et al.

    Ranking management strategies with complex outcomes: an AHP-fuzzy evaluation of recreational fishing using an integrated agent-based model of a coral reef ecosystem

    Environ. Model. Softw.

    (2012)
  • J.R. Kasprzyk et al.

    Many objective robust decision making for complex environmental systems undergoing change

    Environ. Modell. Softw.

    (2013)
  • J.M. Kirby et al.

    Climate change and environmental water reallocation in the Murray–Darling Basin: impacts on flows, diversions and economic returns to irrigation

    J. Hydrol

    (2014)
  • Q. Liu et al.

    A new computational method of a moment-independent uncertainty importance measure

    Reliab. Eng. Syst. Saf.

    (2009)
  • S. Marino et al.

    A methodology for performing global uncertainty and sensitivity analysis in systems biology

    J. Theor. Biol.

    (2008)
  • J. Nossent et al.

    Sobol’ sensitivity analysis of a complex environmental model

    Environ. Modell. Softw.

    (2011)
  • J.E. Oakley et al.

    Simulation sample sizes for Monte Carlo partial EVPI calculations

    J. Health Econ.

    (2010)
  • A. Saltelli et al.

    Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index

    Comput. Phys. Commun.

    (2010)
  • A. Saltelli et al.

    Non-parametric statistics in sensitivity analysis for model output: a comparison of selected techniques

    Reliab. Eng. Syst. Saf.

    (1990)
  • M.-J. Shin et al.

    Addressing ten questions about conceptual rainfall–runoff models with global sensitivity analyses in R

    J. Hydrol.

    (2013)
  • X. Song et al.

    Time-dependent sensitivity of a process-based ecological model

    Ecol. Modell.

    (2013)
  • X. Song et al.

    Variance-based sensitivity analysis of a forest growth model

    Ecol. Modell.

    (2012)
  • C.-S. Zhan et al.

    An efficient integrated approach for global sensitivity analysis of hydrological model parameters

    Environ. Modell. Softw.

    (2013)
  • G. Zhao et al.

    Sensitivity and uncertainty analysis of the APSIM-wheat model: interactions between cultivar, environmental, and management parameters

    Ecol. Modell.

    (2014)
  • D.R. Anderson et al.

    An Introduction to Management Science: Quantitative Approaches to Decision Making

    (2012)
  • I.J. Bateman et al.

    Bringing ecosystem services into economic decision-making: land use in the United Kingdom

    Science

    (2013)
  • L.A. Cox

    Confronting deep uncertainties in risk analysis

    Risk Anal.

    (2012)
  • R.I. Cukier et al.

    Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory

    J. Chem. Phys.

    (1973)
  • Cited by (0)

    View full text