Abstract: The success of first-principles electronic-structure calculation for predictive modeling in chemistry, solid-state physics, and materials science is constrained by the limitations on simulated length scales and timescales due to the computational cost and its scaling. Techniques based on machine-learning ideas for interpolating the Born-Oppenheimer potential energy surface without explicitly describing electrons have recently shown great promise, but accurately and efficiently fitting the physically relevant space of configurations remains a challenging goal. Here, we present a Gaussian approximation potential for silicon that achieves this milestone, accurately reproducing density-functional-theory reference results for a wide range of observable properties, including crystal, liquid, and amorphous bulk phases, as well as point, line, and plane defects. We demonstrate that this new potential enables calculations such as finite-temperature phase-boundary lines, self-diffusivity in the liquid, formation of the amorphous by slow quench, and dynamic brittle fracture, all of which are very expensive with a first-principles method. We show that the uncertainty quantification inherent to the Gaussian process regression framework gives a qualitative estimate of the potential’s accuracy for a given atomic configuration. The success of this model shows that it is indeed possible to create a useful machine-learning-based interatomic potential that comprehensively describes a material on the atomic scale and serves as a template for the development of such models in the future.

Abstract: To assure that an autonomous car is driving safely on public roads, its object detection module should not only work correctly, but show its prediction confidence as well. Previous object detectors driven by deep learning do not explicitly model uncertainties in the neural network. We tackle with this problem by presenting practical methods to capture uncertainties in a 3D vehicle detector for Lidar point clouds. The proposed probabilistic detector represents reliable epistemic uncertainty and aleatoric uncertainty in classification and localization tasks. Experimental results show that the epistemic uncertainty is related to the detection accuracy, whereas the aleatoric uncertainty is influenced by vehicle distance and occlusion. The results also show that we can improve the detection performance by 1%–5% by modeling the aleatoric uncertainty.

Abstract: In ecology, the true causal structure for a given problem is often not known, and several plausible models and thus model predictions exist. It has been claimed that using weighted averages of these models can reduce prediction error, as well as better reflect model selection uncertainty. These claims, however, are often demonstrated by isolated examples. Analysts must better understand under which conditions model averaging can improve predictions and their uncertainty estimates. Moreover, a large range of different model averaging methods exists, raising the question of how they differ in their behaviour and performance. Here, we review the mathematical foundations of model averaging along with the diversity of approaches available. We explain that the error in model‐averaged predictions depends on each model's predictive bias and variance, as well as the covariance in predictions between models, and uncertainty about model weights. We show that model averaging is particularly useful if the predictive error of contributing model predictions is dominated by variance, and if the covariance between models is low. For noisy data, which predominate in ecology, these conditions will often be met. Many different methods to derive averaging weights exist, from Bayesian over information‐theoretical to cross‐validation optimized and resampling approaches. A general recommendation is difficult, because the performance of methods is often context dependent. Importantly, estimating weights creates some additional uncertainty. As a result, estimated model weights may not always outperform arbitrary fixed weights, such as equal weights for all models. When averaging a set of models with many inadequate models, however, estimating model weights will typically be superior to equal weights. We also investigate the quality of the confidence intervals calculated for model‐averaged predictions, showing that they differ greatly in behaviour and seldom manage to achieve nominal coverage. Our overall recommendations stress the importance of non‐parametric methods such as cross‐validation for a reliable uncertainty quantification of model‐averaged predictions.

Abstract: The success of first principles electronic structure calculation for predictive modeling in chemistry, solid state physics, and materials science is constrained by the limitations on simulated length and time scales due to computational cost and its scaling. Techniques based on machine learning ideas for interpolating the Born-Oppenheimer potential energy surface without explicitly describing electrons have recently shown great promise, but accurately and efficiently fitting the physically relevant space of configurations has remained a challenging goal. Here we present a Gaussian Approximation Potential for silicon that achieves this milestone, accurately reproducing density functional theory reference results for a wide range of observable properties, including crystal, liquid, and amorphous bulk phases, as well as point, line, and plane defects. We demonstrate that this new potential enables calculations that would be extremely expensive with a first principles electronic structure method, such as finite temperature phase boundary lines, self-diffusivity in the liquid, formation of the amorphous by slow quench, and dynamic brittle fracture. We show that the uncertainty quantification inherent to the Gaussian process regression framework gives a qualitative estimate of the potential's accuracy for a given atomic configuration. The success of this model shows that it is indeed possible to create a useful machine-learning-based interatomic potential that comprehensively describes a material, and serves as a template for the development of such models in the future.

Abstract: Uncertainty quantification plays a critical role in the process of decision making and optimization in many fields of science and engineering. The field has gained an overwhelming attention among researchers in recent years resulting in an arsenal of different methods. Probabilistic forecasting and in particular prediction intervals (PIs) are one of the techniques most widely used in the literature for uncertainty quantification. Researchers have reported studies of uncertainty quantification in critical applications such as medical diagnostics, bioinformatics, renewable energies, and power grids. The purpose of this survey paper is to comprehensively study neural network-based methods for construction of prediction intervals. It will cover how PIs are constructed, optimized, and applied for decision-making in presence of uncertainties. Also, different criteria for unbiased PI evaluation are investigated. The paper also provides some guidelines for further research in the field of neural network-based uncertainty quantification.

Abstract: This paper considers the generation of prediction intervals (PIs) by neural networks for quantifying uncertainty in regression tasks. It is axiomatic that high-quality PIs should be as narrow as possible, whilst capturing a specified portion of data. We derive a loss function directly from this axiom that requires no distributional assumption. We show how its form derives from a likelihood principle, that it can be used with gradient descent, and that model uncertainty is accounted for in ensembled form. Benchmark experiments show the method outperforms current state-of-the-art uncertainty quantification methods, reducing average PI width by over 10%.

Abstract: Modern imaging methods rely strongly on Bayesian inference techniques to solve challenging imaging problems. Currently, the predominant Bayesian computation approach is convex optimization, which scales very efficiently to high-dimensional image models and delivers accurate point estimation results. However, in order to perform more complex analyses, for example, image uncertainty quantification or model selection, it is necessary to use more computationally intensive Bayesian computation techniques such as Markov chain Monte Carlo methods. This paper presents a new and highly efficient Markov chain Monte Carlo methodology to perform Bayesian computation for high-dimensional models that are log-concave and nonsmooth, a class of models that is central in imaging sciences. The methodology is based on a regularized unadjusted Langevin algorithm that exploits tools from convex analysis, namely, Moreau--Yoshida envelopes and proximal operators, to construct Markov chains with favorable convergence properties. ...

Abstract: Background: The mass, or binding energy, is the basis property of the atomic nucleus. It determines its stability and reaction and decay rates. Quantifying the nuclear binding is important for understanding the origin of elements in the universe. The astrophysical processes responsible for the nucleosynthesis in stars often take place far from the valley of stability, where experimental masses are not known. In such cases, missing nuclear information must be provided by theoretical predictions using extreme extrapolations. To take full advantage of the information contained in mass model residuals, i.e., deviations between experimental and calculated masses, one can utilize Bayesian machine-learning techniques to improve predictions. Purpose: To improve the quality of model-based predictions of nuclear properties of rare isotopes far from stability, we consider the information contained in the residuals in the regions where the experimental information exist. As a case in point, we discuss two-neutron separation energies S2n of even-even nuclei. Through this observable, we assess the predictive power of global mass models towards more unstable neutron-rich nuclei and provide uncertainty quantification of predictions. Methods: We consider 10 global models based on nuclear density functional theory with realistic energy density functionals as well as two more phenomenological mass models. The emulators of S2n residuals and credibility intervals (Bayesian confidence intervals) defining theoretical error bars are constructed using Bayesian Gaussian processes and Bayesian neural networks. We consider a large training dataset pertaining to nuclei whose masses were measured before 2003. For the testing datasets, we considered those exotic nuclei whose masses have been determined after 2003. By establishing statistical methodology and parameters, we carried out extrapolations toward the 2n dripline. Results: While both Gaussian processes and Bayesian neural networks reduce the root-mean-square (rms) deviation from experiment significantly, GP offers a better and much more stable performance. The increase in the predictive power of microscopic models aided by the statistical treatment is quite astonishing: The resulting rms deviations from experiment on the testing dataset are similar to those of more phenomenological models. We found that Bayesian neural networks results are prone to instabilities caused by the large number of parameters in this method. Moreover, since the classical sigmoid activation function used in this approach has linear tails that do not vanish, it is poorly suited for a bounded extrapolation. The empirical coverage probability curves we obtain match very well the reference values, in a slightly conservative way in most cases, which is highly desirable to ensure honesty of uncertainty quantification. The estimated credibility intervals on predictions make it possible to evaluate predictive power of individual models and also make quantified predictions using groups of models. Conclusions: The proposed robust statistical approach to extrapolation of nuclear model results can be useful for assessing the impact of current and future experiments in the context of model developments. The new Bayesian capability to evaluate residuals is also expected to impact research in the domains where experiments are currently impossible, for instance, in simulations of the astrophysical r process.

Abstract: Surrogate strategies are used widely for uncertainty quantification of groundwater models in order to improve computational efficiency. However, their application to dynamic multiphase flow problems is hindered by the curse of dimensionality, the saturation discontinuity due to capillarity effects, and the time-dependence of the multi-output responses. In this paper, we propose a deep convolutional encoder-decoder neural network methodology to tackle these issues. The surrogate modeling task is transformed to an image-to-image regression strategy. This approach extracts high-level coarse features from the high-dimensional input permeability images using an encoder, and then refines the coarse features to provide the output pressure/saturation images through a decoder. A training strategy combining a regression loss and a segmentation loss is proposed in order to better approximate the discontinuous saturation field. To characterize the high-dimensional time-dependent outputs of the dynamic system, time is treated as an additional input to the network that is trained using pairs of input realizations and of the corresponding system outputs at a limited number of time instances. The proposed method is evaluated using a geological carbon storage process-based multiphase flow model with a 2500-dimensional stochastic permeability field. With a relatively small number of training data, the surrogate model is capable of accurately characterizing the spatio-temporal evolution of the pressure and discontinuous CO2 saturation fields and can be used efficiently to compute the statistics of the system responses.

Abstract: Traditional structural uncertainty analysis is mainly based on probability models and requires the establishment of accurate parametric probability distribution functions using large numbers of experimental samples. In many actual engineering problems, the probability distributions of some parameters can be established due to sufficient samples available, whereas for some parameters, due to the lack or poor quality of samples, only their variation intervals can be obtained, or their probability distribution types can be determined based on the existing data while some of the distribution parameters such as mean and standard deviation can only be given interval estimations. This thus will constitute an important type of probability-interval hybrid uncertain problem, in which the aleatory and epistemic uncertainties both exist. The probability-interval hybrid uncertainty analysis provides an important mean for reliability analysis and design of many complex structures, and has become one of the research focuses in the field of structural uncertainty analysis over the past decades. This paper reviews the four main research directions in this area, i.e., uncertainty modeling, uncertainty propagation analysis, structural reliability analysis, and reliability-based design optimization. It summarizes the main scientific problems, technical difficulties, and current research status of each direction. Based on the review, this paper also provides an outlook for future research in probability-interval hybrid uncertainty analysis.

Abstract: An open-source, scalable and model-independent (non-intrusive) implementation of an iterative ensemble smoother has been developed to alleviate the computational burden associated with history-matching and uncertainty quantification of real-world-scale environmental models that have very high dimensional parameter spaces. The tool, named pestpp-ies, implements the ensemble-smoother form of the popular Gauss-Levenberg-Marquardt algorithm, uses the pest model-interface protocols and includes a built-in parallel run manager, multiple lambda testing and model run failure tolerance. As a demonstration of its capabilities, pestpp-ies is applied to a synthetic groundwater model with thousands of parameters and to a real-world groundwater flow and transport model with tens of thousands of parameters. pestpp-ies is shown to efficiently and effectively condition parameters in both cases and can provide means to estimate posterior forecast uncertainty when the forecasts depend on large numbers of parameters.

Abstract: Understanding the uncertainty of a neural network's (NN) predictions is essential for many purposes. The Bayesian framework provides a principled approach to this, however applying it to NNs is challenging due to large numbers of parameters and data. Ensembling NNs provides an easily implementable, scalable method for uncertainty quantification, however, it has been criticised for not being Bayesian. This work proposes one modification to the usual process that we argue does result in approximate Bayesian inference; regularising parameters about values drawn from a distribution which can be set equal to the prior. A theoretical analysis of the procedure in a simplified setting suggests the recovered posterior is centred correctly but tends to have an underestimated marginal variance, and overestimated correlation. However, two conditions can lead to exact recovery. We argue that these conditions are partially present in NNs. Empirical evaluations demonstrate it has an advantage over standard ensembling, and is competitive with variational methods.

Abstract: Advances in manufacturing process technology are key ensembles for the production of integrated circuits in the sub-micrometer region. It is of paramount importance to assess the effects of tolerances in the manufacturing process on the performance of modern integrated circuits. The polynomial chaos expansion has emerged as a suitable alternative to standard Monte Carlo-based methods that are accurate, but computationally cumbersome. This paper provides an overview of the most recent developments and challenges in the application of polynomial chaos-based techniques for uncertainty quantification in integrated circuits, with particular focus on high-dimensional problems.

Abstract: In the quest for advanced propulsion and power-generation systems, high-fidelity simulations are too computationally expensive to survey the desired design space, and a new design methodology is needed that combines engineering physics, computer simulations, and statistical modeling. In this article, we propose a new surrogate model that provides efficient prediction and uncertainty quantification of turbulent flows in swirl injectors with varying geometries, devices commonly used in many engineering applications. The novelty of the proposed method lies in the incorporation of known physical properties of the fluid flow as simplifying assumptions for the statistical model. In view of the massive simulation data at hand, which is on the order of hundreds of gigabytes, these assumptions allow for accurate flow predictions in around an hour of computation time. To contrast, existing flow emulators which forgo such simplifications may require more computation time for training and prediction than is n...

Abstract: Computational models in neuroscience typically contain many parameters that are poorly constrained by experimental data. Uncertainty quantification and sensitivity analysis provide rigorous procedures to quantify how the model output depends on this parameter uncertainty. Unfortunately, the application of such methods is not yet standard within the field of neuroscience. Here we present Uncertainpy, an open-source Python toolbox, tailored to perform uncertainty quantification and sensitivity analysis of neuroscience models. Uncertainpy aims to make it quick and easy to get started with uncertainty analysis, without any need for detailed prior knowledge. The toolbox allows uncertainty quantification and sensitivity analysis to be performed on already existing models without needing to modify the model equations or model implementation. Uncertainpy bases its analysis on polynomial chaos expansions, which are more efficient than the more standard Monte-Carlo based approaches. Uncertainpy is tailored for neuroscience applications by its built-in capability for calculating characteristic features in the model output. The toolbox does not merely perform a point-to-point comparison of the ``raw'' model output (e.g., membrane voltage traces), but can also calculate the uncertainty and sensitivity of salient model response features such as spike timing, action potential width, average interspike interval, and other features relevant for various neural and neural network models. Uncertainpy comes with several common models and features built in, and including custom models and new features is easy. The aim of the current paper is to present Uncertainpy to the neuroscience community in a user-oriented manner. To demonstrate its broad applicability, we perform an uncertainty quantification and sensitivity analysis of three case studies relevant for neuroscience: the original Hodgkin-Huxley point-neuron model for action potential generation, a multi-compartmental model of a thalamic interneuron implemented in the NEURON simulator, and a sparsely connected recurrent network model implemented in the NEST simulator.

Abstract: Measuring system reliability by a reasonable metric is a common problem in reliability engineering. Since real systems are usually uncertain random systems affected by both aleatory and epistemic uncertainties, existing reliability metrics are unreliable. This paper proposes a general reliability metric, called belief reliability metric, to cope with the problem. In this paper, the belief reliability is defined as the chance that a system state is within a feasible domain. Mathematically, the metric can degenerate to either probability theory-based reliability, which copes with aleatory uncertainty, or uncertainty theory-based reliability, which considers the effect of epistemic uncertainty. Based on the proposed metric, some commonly used belief reliability indexes, such as belief reliability distribution, mean time to failure, and belief reliable life, are introduced. We also develop system belief reliability formulas for different systems configurations. To further illustrate the formulas, a real case study is performed.

Abstract: The atmospheric component of Energy Exascale Earth System Model (E3SM) version 1 (EAMv1) has included many new features in the physics parameterizations compared to its predecessors. Potential complex nonlinear interactions among the new features create a significant challenge for understanding the model behaviors and parameter tuning. Using the one-at-a-time method, the benefit of tuning one parameter may offset the benefit of tuning another parameter, or improvement in one target variable may lead to degradation in another target variable. To better understand the EAMv1 model behaviors and physics, we conducted a large number of short simulations (3 days) in which 18 parameters carefully selected from parameterizations of deep convection, shallow convection and cloud macrophysics and microphysics were perturbed simultaneously using the Latin Hypercube sampling method. From the Perturbed Parameters Ensemble (PPE) simulations and use of different skill score functions, we identified the most sensitive parameters, quantified how the model responds to changes of the parameters for both global mean and spatial distribution, and estimated the maximum likelihood of model parameter space for a number of important fidelity metrics. Comparison of the parametric sensitivity using simulations of two different lengths suggests that PPE using short simulations has some bearing on understanding parametric sensitivity of longer simulations. Results from this analysis provide a more comprehensive picture of the EAMv1 behavior. The difficulty in reducing biases in multiple variables simultaneously highlights the need of characterizing model structural uncertainty (so-called embedded errors) to inform future development efforts.

Abstract: Weather forecasting is usually solved through numerical weather prediction (NWP), which can sometimes lead to unsatisfactory performance due to inappropriate setting of the initial states. In this paper, we design a data-driven method augmented by an effective information fusion mechanism to learn from historical data that incorporates prior knowledge from NWP. We cast the weather forecasting problem as an end-to-end deep learning problem and solve it by proposing a novel negative log-likelihood error (NLE) loss function. A notable advantage of our proposed method is that it simultaneously implements single-value forecasting and uncertainty quantification, which we refer to as deep uncertainty quantification (DUQ). Efficient deep ensemble strategies are also explored to further improve performance. This new approach was evaluated on a public dataset collected from weather stations in Beijing, China. Experimental results demonstrate that the proposed NLE loss significantly improves generalization compared to mean squared error (MSE) loss and mean absolute error (MAE) loss. Compared with NWP, this approach significantly improves accuracy by 47.76%, which is a state-of-the-art result on this benchmark dataset. The preliminary version of the proposed method won 2nd place in an online competition for daily weather forecasting.

Abstract: Multifidelity surrogates (MFS) combine low-fidelity models with few high-fidelity samples to infer the response of the high-fidelity model for design optimization or uncertainty quantification. Mos...