RBM models reveal how hidden unit tail behavior affects pattern reconstruction.
problem Understanding how the tail behavior of hidden units in RBMs influences pattern reconstruction.
method Identified an effective energy function for RBMs and studied its local minima.
result The ability to reconstruct patterns depends on the tail behavior of the hidden unit prior distribution.
Study on hidden units in finite Bayesian neural networks and their tail properties.
problem Understanding the behavior of hidden units in finite Bayesian neural networks.
method Introduced a generalized Weibull-tail property to describe hidden units tails.
result Unit priors become heavier-tailed going deeper, providing insights into finite Bayesian neural networks.
We study layered neural networks of rectified linear units (ReLU) in a modelling framework for stochastic training processes. The comparison with sigmoidal activation functions is in the center of interest. We compute typical learning curves for shallow networks with K hidden units in matching student teacher scenarios…
Lipschitz RNNs improve stability and performance in various tasks.
problem Improving stability and performance of RNNs.
method Introduced a Lipschitz recurrent unit with a linear and Lipschitz nonlinear component for stability analysis.
result Lipschitz RNNs outperform existing units on benchmark tasks.
Randomly chosen primary hidden units and derived secondary units reduce neural network complexity.
problem Large number of hidden units in neural networks.
method Introducing primary and secondary hidden units with random weights for primary units and derived weights for secondary units.
result Significant reduction in the number of hidden units without compromising accuracy.
The hidden tail of empirical distributions is analyzed using extreme value theory.
problem Understanding the bias between in-sample mean and true statistical mean for large n. method Extreme value theory applied to empirical distributions and their moments.
result The hidden moment of order 0 for power law distributions follows an exponential distribution with expectation 1/n. Bayesian neural networks ignore data in infinite units limit.
problem Pathological behavior of posterior in over-parameterized networks.
method Mean-field variational inference in infinite hidden units limit.
result Posterior mean converges to zero, ignoring data.
Study tail risk aggregation under dependence uncertainty.
problem Risk aggregation under dependence uncertainty and hidden dependence.
method Introduce hidden dependence, show compatibility with small perturbations, quantify portfolio risk.
result Small deviations in dependence structure can lead to significant risk underestimation.
Study examines dependence properties of Bayesian neural network units in finite-width networks.
problem Understanding dependence properties of hidden units in practical finite-width Bayesian neural networks.
method Theoretical analysis and empirical evaluation of depth and width impacts.
result Hidden units in finite-width Bayesian neural networks are dependent, contrary to the infinite-width limit assumption.
Multivariate regular variation plays a role assessing tail risk in diverse applications such as finance, telecommunications, insurance and environmental science. The classical theory, being based on an asymptotic model, sometimes leads to inaccurate and useless estimates of probabilities of joint tail regions. This pro…
Deep belief networks are a powerful way to model complex probability distributions. However, learning the structure of a belief network, particularly one with hidden units, is difficult. The Indian buffet process has been used as a nonparametric Bayesian prior on the directed structure of a belief network with a single…
Deep belief networks can approximate any multivariate density with binary hidden units.
problem Approximating multivariate probability densities with binary hidden units.
method Sharp quantitative bounds on approximation error in terms of hidden units.
result Deep belief networks can approximate any multivariate density with binary hidden units under mild integrability requirements.
We establish upper bounds for the minimal number of hidden units for which a binary stochastic feedforward network with sigmoid activation probabilities and a single hidden layer is a universal approximator of Markov kernels. We show that each possible probabilistic assignment of the states of n output units, given t…
Since learning is typically very slow in Boltzmann machines, there is a need to restrict connections within hidden layers. However, the resulting states of hidden units exhibit statistical dependencies. Based on this observation, we propose using l1/l2 regularization upon the activation possibilities of hidden unit…
Stochastic Volatility in Mean models with heavy-tailed distributions using Hidden Markov Models
problem Accurate inference for Stochastic Volatility in Mean models with heavy-tailed distributions
method Numerically stable estimation procedure and parallel computing
result Significant reduction in computational times
Econometric framework integrates heavy-tailed distributions with behavioral probability weighting for better asset pricing.
problem Underestimation of Value-at-Risk by traditional models in asset pricing.
method Developed an econometric framework combining heavy-tailed Student's t distributions with behavioral probability weighting. result Student's t specifications outperform Gaussian models in 88.4% of cases, reducing underestimation of Value-at-Risk by 16.5 percentage points. New MBL hidden Born machine learns various tasks.
problem Learning from quantum many-body systems.
method MBL dynamics and hidden units for training.
result Enhanced trainability and stability in learning.
Hidden regular variation is a sub-model of multivariate regular variation and facilitates accurate estimation of joint tail probabilities. We generalize the model of hidden regular variation to what we call hidden domain of attraction. We exhibit examples that illustrate the need for a more general model and discuss de…
New term ADS describes how machine learning can change user behavior.
problem Machine learning systems can unintentionally change user behavior, affecting performance.
method Introduced `unit tests` and mitigation strategy for hidden incentives in auto-induced distributional shift.
result Meta-learning and Q-learning sometimes fail unit tests but pass with mitigation strategy.
A neural network with a single hidden layer can't represent certain multivariable functions.
problem Representing certain multivariable functions with a neural network having only one hidden layer.
method Developed a continuum version of a one-hidden-layer neural network with ReLU activation, and proved constraints on its parameters and second derivative.
result Existence of a smooth binary function that cannot be precisely represented by any such neural network.
PH-VAE models heavy-tailed data with flexible Phase-Type distributions.
problem Standard VAEs fail to capture heavy-tailed behavior in real-world data.
method PH-VAE uses Phase-Type distributions defined by continuous-time Markov chains to adaptively model tail behavior.
result PH-VAE significantly outperforms existing heavy-tail-aware VAEs in approximating diverse heavy-tailed distributions.
Study compares Bitcoin and Ethereum tail behavior using Q-Q plots.
problem Examining tail risk in cryptocurrency returns.
method Used Q-Q plots and Generalized Tempered Stable (GTS) distribution.
result Ethereum shows more extreme values than Bitcoin, indicating greater tail risk.
The problem of attributing a deep network's prediction to its \emph{input/base} features is well-studied. We introduce the notion of \emph{conductance} to extend the notion of attribution to the understanding the importance of \emph{hidden} units. Informally, the conductance of a hidden unit of a deep network is the \e…
Deep learning is the state-of-the-art in fields such as visual object recognition and speech recognition. This learning uses a large number of layers, huge number of units, and connections. Therefore, overfitting is a serious problem. To avoid this problem, dropout learning is proposed. Dropout learning neglects some i…
Paper improves normalizing flows to better capture distribution tails.
problem Difficult to learn tail behavior of distributions.
method Develops a new type of flows using flexible base distributions and data-driven linear layers.
result Improves accuracy, especially on distribution tails, and generates heavy-tailed data.
The infinite restricted Boltzmann machine (iRBM) is an extension of the classic RBM. It enjoys a good property of automatically deciding the size of the hidden layer according to specific training data. With sufficient training, the iRBM can achieve a competitive performance with that of the classic RBM. However, the c…
We study tilting subweibull distributions and their tail behavior.
problem Understanding tail behavior of subweibull distributions.
method Alternative characterizations and conditions for tail behavior preservation.
result Conditions for tail behavior preservation after exponential tilting.
Permutation of any two hidden units yields invariant properties in typical deep generative neural networks. This permutation symmetry plays an important role in understanding the computation performance of a broad class of neural networks with two or more hidden units. However, a theoretical study of the permutation sy…
New insights into tail behavior of heavy-tailed random vectors and processes.
problem Understanding tail behavior of aggregates of heavy-tailed random vectors.
method Analyzing multivariate regularly varying random vectors and Lévy processes.
result More than one large jump can determine tail behavior of aggregates.
New method reduces overfitting in deep neural networks by measuring and regulating hidden unit diversity.
problem Overfitting in deep neural networks.
method Introduces a new redundancy measure based on mutual information to improve generalization.
result Reduction of redundancy improves generalization capacity, reducing overfitting.
Study tail risk in high-frequency finance using L1-regularized regression.
problem Measuring tail risk dynamics in high-frequency financial markets.
method Dynamic extreme value regression model with L1-regularized maximum likelihood estimator. result Severity of extreme losses well predicted by low price impact in high volatility periods.
The paper examines how heavy-tailed risks behave under Gaussian copula models.
problem Understanding tail risk probabilities with heavy-tailed marginal risks and Gaussian dependence.
method Modeling heavy-tailed risks using regular variation and analyzing tail probabilities under Gaussian copula.
result The rate of decay of tail set probabilities varies with the type of tail sets and Gaussian correlation matrix.
To infer a multilayer representation of high-dimensional count vectors, we propose the Poisson gamma belief network (PGBN) that factorizes each of its layers into the product of a connection weight matrix and the nonnegative real hidden units of the next layer. The PGBN's hidden layers are jointly trained with an upwar…
A new algebra for probabilistic programming improves tail behavior accuracy.
problem Inaccurate tail behavior in probabilistic models based on neural networks.
method Developed a three-parameter tail asymptotics algebra based on the generalized Gamma distribution.
result Inference algorithms using the heavy-tailed algebra achieve superior performance.
A novel quantum model improves RBM performance and is efficiently trainable.
problem Improving the performance of RBM models.
method Quantum model with parametrically coupled fermions to classical signals.
result The model outperforms classical RBM with the same number of hidden units.
Structural RBM reduces parameters for image denoising and classification.
problem High parameter count in RBMs limits their applicability to large datasets.
method Introduces SRBM with constrained connections to reduce parameters.
result SRBM achieves better performance and faster training than vanilla RBM.
Studied how heavy-tailed behavior affects SGD's generalization in quadratic optimization.
problem Link between heavy-tailed behavior and generalization in SGD.
method Used heavy-tailed stochastic differential equation and proved stability bounds.
result Stability of SGD depends on the loss function's tail behavior.
Revisits granular models explaining firm growth rates and sizes.
problem Understanding the relationship between firm size and growth rate statistics.
method Developed new theoretical insights linking firm size and growth rate statistics within granular models.
result Growth volatility distribution is size-independent but fat-tailed, challenging granular models.
New models explain heavy-tailed behavior in neural networks.
problem Heavy-tailed spectral densities in neural networks.
method High-temperature Marchenko-Pastur (HTMP) ensemble models.
result Heavy-tailed behavior arises from three factors: data structure, training temperature, and eigenvector entropy.
High-dimensional shrinkage risk depends on the default prior for the common scale.
problem Choosing the default prior for the common scale in high-dimensional shrinkage.
method Using radial-power benchmark to compare variance-flat and standard deviation-flat priors.
result The standard deviation-flat prior has a one-unit asymptotic risk advantage near the origin.
Neural networks can approximate any continuous function with a single hidden layer.
problem Approximating any continuous function using neural networks.
method Direct algebraic proof and explicit quantification of hidden units required.
result Explicit bounds on the number of hidden units needed for approximation.
Study tail behavior of sum of heavy-tailed risks with copulas.
problem Analyzing the tail behavior of sums of heavy-tailed risks with dependence modeled by copulas.
method Modeling dependence with copulas and analyzing tail asymptotics of sums of heavy-tailed risks.
result Obtained asymptotic expansions for Value-at-Risk of aggregate risk.
Direct measurements of Gini coefficients by conventional arithmetic calculations are a poor estimator, even if paradoxically, they include the entire population, as because of super-additivity they cannot lend themselves to comparisons between units of different size, and intertemporal analyses are vitiated by the popu…
Stochastic binary hidden units in a multi-layer perceptron (MLP) network give at least three potential benefits when compared to deterministic MLP networks. (1) They allow to learn one-to-many type of mappings. (2) They can be used in structured prediction problems, where modeling the internal structure of the output i…
The goal of this paper is to study organized flocking behavior and systemic risk in heterogeneous mean-field interacting diffusions. We illustrate in a number of case studies the effect of heterogeneity in the behavior of systemic risk in the system, i.e., the risk that several agents default simultaneously as a result…
The standard deviation and Gini mean difference order based on tail behavior.
problem Ordering between standard deviation and Gini mean difference for real-valued risks.
method Analysis of the mean excess function of the pairwise difference ∣X−X′∣. result Dominance regimes of SD and GMD are determined by tail behavior of the distribution.
This work tackles the problem of learning a set of language specific acoustic units from unlabeled speech recordings given a set of labeled recordings from other languages. Our approach may be described by the following two steps procedure: first the model learns the notion of acoustic units from the labelled data and …
State of the art Deep Neural Networks (DNN) can now achieve above human level accuracy on image classification tasks. However their outstanding performances come along with a complex inference mechanism making them arduously interpretable models. In order to understand the underlying prediction rules of DNNs, Dhamdhere…