This paper provides mathematical foundations for regression methods used in forward initial margin approximation.
problem Developing robust methods for approximating forward initial margin.
method Introduces mathematical rigor to show that regression methods are variations of approximating the conditional expectation function.
result Each regression method is a numerical estimation of the conditional expectation with a different functional form.
Neural networks can approximate high-dimensional classifiers with ReLU networks under margin conditions.
problem Approximating high-dimensional discontinuous classifiers with neural networks.
method Using ReLU neural networks with three hidden layers, approximating a classifier with a Barron-regular decision boundary.
result High-dimensional discontinuous classifiers can be approximated with a rate of n−1 under strong margin conditions. We address the problem of learning the parameters in graphical models when inference is intractable. A common strategy in this case is to replace the partition function with its Bethe approximation. We show that there exists a regime of empirical marginals where such Bethe learning will fail. By failure we mean that th…
Due to the intractable partition function, the exact likelihood function for a Markov random field (MRF), in many situations, can only be approximated. Major approximation approaches include pseudolikelihood and Laplace approximation. In this paper, we propose a novel way of approximating the likelihood function throug…
Improves hyperparameter learning in GP models with non-conjugate likelihoods.
problem Hyperparameter learning entangled with approximate inference in GP models.
method Hybrid training procedure combining VI for inference and EP-like marginal likelihood approximation for hyperparameter learning.
result Empirically demonstrates the effectiveness of the proposed training procedure across various data sets.
New probabilistic complexity measures for linear and kernel methods.
problem Limitations of linear and kernel methods in machine learning.
method Introducing approximate notions of dimensional and margin complexity.
result Approximate complexity measures are both sufficient and necessary for learning.
Bayesian inference in the presence of an intractable likelihood function is computationally challenging. When following a Markov chain Monte Carlo (MCMC) approach to approximate the posterior distribution in this context, one typically either uses MCMC schemes which target the joint posterior of the parameters and some…
We introduce a useful tool for analyzing boosting algorithms called the ``smooth margin function,'' a differentiable approximation of the usual margin for boosting algorithms. We present two boosting algorithms based on this smooth margin, ``coordinate ascent boosting'' and ``approximate coordinate ascent boosting,'' w…
Bayesian approach learns invariances from data alone, but last layer approximation is not always sufficient.
problem Learning invariances in neural networks using only training data.
method Bayesian marginal likelihood for last layer, custom optimisation routine, new lower bound.
result Partial success on standard benchmarks and medical imaging dataset, failure on CIFAR10.
We present and implement two algorithms for analytic asymptotic evaluation of the marginal likelihood of data given a Bayesian network with hidden nodes. As shown by previous work, this evaluation is particularly hard for latent Bayesian network models, namely networks that include hidden variables, where asymptotic ap…
Improved likelihood-free inference by localizing and refining low-dimensional approximations.
problem Poor performance of common likelihood-free methods in high-dimensional models.
method Localisation followed by refinement of low-dimensional summaries.
result Improved accuracy in marginal posteriors through localized and refined approximations.
Graphical models trained using maximum likelihood are a common tool for probabilistic inference of marginal distributions. However, this approach suffers difficulties when either the inference process or the model is approximate. In this paper, the inference process is first defined to be the minimization of a convex f…
Deep Gaussian processes provide a flexible approach to probabilistic modelling of data using either supervised or unsupervised learning. For tractable inference approximations to the marginal likelihood of the model must be made. The original approach to approximate inference in these models used variational compressio…
New MKABSDEs help calculate initial margins in financial contracts.
problem Calculating initial margins in financial contracts with dependencies.
method Introduced MKABSDEs, provided existence and uniqueness, applied to CVaR, used deterministic and Monte-Carlo methods for numerical approximations.
result MKABSDEs provide a new way to solve for initial margins in financial contracts.
New method controls error in low-dimensional marginals of spatial models.
problem Inaccurate approximation of low-dimensional marginals in spatial models.
method Stein's method with δ-locality condition for spatial models.
result Uniform error bound for marginals of approximate distributions.
Improves Gaussian process regression without bias.
problem Bias in Gaussian process regression estimates.
method Adaptive computation selection to minimize bias.
result Guaranteed small bias in log marginal likelihood estimates.
A new diffusion method approximates Schrödinger bridge with improved convergence.
problem Approximating Schrödinger bridge with Langevin diffusion.
method Leveraging Langevin diffusion to approximate Schrödinger bridge.
result The difference between the two approximations is proportional to the score function.
Inference methods are often formulated as variational approximations: these approximations allow easy evaluation of statistics by marginalization or linear response, but these estimates can be inconsistent. We show that by introducing constraints on covariance, one can ensure consistency of linear response with the var…
Corrects errors in ILA for Bayesian inference in LGMs.
problem Error in ILA for non-Gaussian likelihoods in LGMs.
method Importance sampling scheme to correct ILA errors.
result Corrected posterior converges to the true posterior with increased samples.
Bayesian inference in state-space models is challenging due to high-dimensional state trajectories. A viable approach is particle Markov chain Monte Carlo, combining MCMC and sequential Monte Carlo to form "exact approximations" to otherwise intractable MCMC methods. The performance of the approximation is limited to t…
We discuss Bayesian methods for learning Bayesian networks when data sets are incomplete. In particular, we examine asymptotic approximations for the marginal likelihood of incomplete data given a Bayesian network. We consider the Laplace approximation and the less accurate but more efficient BIC/MDL approximation. We …
Marginal MAP problems are notoriously difficult tasks for graphical models. We derive a general variational framework for solving marginal MAP problems, in which we apply analogues of the Bethe, tree-reweighted, and mean field approximations. We then derive a "mixed" message passing algorithm and a convergent alternati…
Gaussian random vectors exhibit the loss of dimension phenomena, which relate to their joint survival tail behaviour. Besides, the fact that the components of such vectors are light-tailed complicates the approximations of various multivariate risk measures significantly. In this contribution we derive precise approxim…
New method for efficient marginalization of discrete latent variables in neural networks.
problem Computational challenges in training models with discrete latent variables.
method Parameterizing discrete distributions using sparse mappings (sparsemax and structured variants) to reduce support and enable efficient marginalization.
result Achieved good performance in various tasks with efficient and practical training.
Belief Propagation has been widely used for marginal inference, however it is slow on problems with large-domain variables and high-order factors. Previous work provides useful approximations to facilitate inference on such models, but lacks important anytime properties such as: 1) providing accurate and consistent mar…
In this paper, we introduce a new form of amortized variational inference by using the forward KL divergence in a joint-contrastive variational loss. The resulting forward amortized variational inference is a likelihood-free method as its gradient can be sampled without bias and without requiring any evaluation of eith…
The paper explores intersectional fairness in machine learning, proving bounds on it.
problem Intersectional fairness in machine learning, especially when multiple protected attributes are involved.
method Statistical analysis and bounds on intersectional fairness, leveraging marginal fairness.
result Theoretical bounds on intersectional fairness can be computed from marginal fairness and other statistical quantities.
Stacked conformal prediction simplifies model validation.
problem Validating stacked predictive models efficiently.
method Meta-learner at the top of a stacked ensemble for approximate marginal validity.
result The method achieves approximate marginal validity without a separate calibration sample.
Neural models improve GLMMs for complex data.
problem Nonlinear relationships in grouped data.
method Replaced linear function with neural networks.
result Improved performance on synthetic and real-world data.
Marginal MAP inference involves making MAP predictions in systems defined with latent variables or missing information. It is significantly more difficult than pure marginalization and MAP tasks, for which a large class of efficient and convergent variational algorithms, such as dual decomposition, exist. In this work,…
Graphical models use graphs to compactly capture stochastic dependencies amongst a collection of random variables. Inference over graphical models corresponds to finding marginal probability distributions given joint probability distributions. In general, this is computationally intractable, which has led to a quest fo…
Several algorithms for solving constraint satisfaction problems are based on survey propagation, a variational inference scheme used to obtain approximate marginal probability estimates for variable assignments. These marginals correspond to how frequently each variable is set to true among satisfying assignments, and …
Optimal Transport (OT) problems arise in a wide range of applications, from physics to economics. Getting numerical approximate solution of these problems is a challenging issue of practical importance. In this work, we investigate the relaxation of the OT problem when the marginal constraints are replaced by some mome…
Bayesian approach sparsifies neural networks efficiently.
problem Efficiently pruning neural networks to save resources.
method Sparsifiability via the Marginal likelihood (SpaM) framework.
result Prunes neural networks effectively without significant loss in performance.
The paper develops a new model-free formula for option initial margins.
problem Calculating initial margins for option portfolios is complex and risky.
method The authors derive a new approximation formula for VaR without assuming a model.
result The new formula performs better than existing methods in simulations.
Multi-class classification with a very large number of classes, or extreme classification, is a challenging problem from both statistical and computational perspectives. Most of the classical approaches to multi-class classification, including one-vs-rest or multi-class support vector machines, require the exact estima…
We study the problem of {\em properly} learning large margin halfspaces in the agnostic PAC model. In more detail, we study the complexity of properly learning d-dimensional halfspaces on the unit ball within misclassification error α⋅OPTγ+ε, where OPTγ is the optimal γ-margin error r…
We speed up marginal inference by ignoring factors that do not significantly contribute to overall accuracy. In order to pick a suitable subset of factors to ignore, we propose three schemes: minimizing the number of model factors under a bound on the KL divergence between pruned and full models; minimizing the KL dive…
New method estimates marginal likelihood for deep learning models using training data alone.
problem Estimation difficulties in marginal likelihood for model selection in deep learning.
method Scalable marginal likelihood estimation based on Laplace's method and Gauss-Newton approximations.
result Estimate outperforms cross-validation and manual tuning on various datasets.
Study online learning of neural networks with margin condition.
problem Online learning of neural networks with sign activation function.
method Characterized margin condition for online learnability, proved mistake bounds, constructed counterexamples.
result Proved optimal mistake bounds and lower bounds for neural networks.
Combines VI and EP for better Gaussian process hyperparameter learning.
problem Improving hyperparameter learning in Gaussian processes for better performance.
method Hybrid training procedure combining Variational Inference (VI) for posterior inference and Expectation Propagation (EP) for hyperparameter learning.
result The hybrid training procedure provides a better learning objective and generalizes better than using only VI or EP.
We propose a new localized inference algorithm for answering marginalization queries in large graphical models with the correlation decay property. Given a query variable and a large graphical model, we define a much smaller model in a local region around the query variable in the target model so that the marginal dist…
This work tackles Bayesian neural networks by addressing loss landscape symmetries.
problem Understanding and optimizing the loss landscape of Bayesian neural networks.
method The approach involves extending marginalized loss barrier formalism to BNNs, proposing a matching algorithm to search for linearly connected solutions using permutation matrices and combinatorial optimization.
result Nearly zero marginalized loss barriers for linearly connected solutions were found.
We introduce a globally-convergent algorithm for optimizing the tree-reweighted (TRW) variational objective over the marginal polytope. The algorithm is based on the conditional gradient method (Frank-Wolfe) and moves pseudomarginals within the marginal polytope through repeated maximum a posteriori (MAP) calls. This m…
A fast method estimates group-adaptive elastic net penalties using co-data.
problem Computational inefficiency in estimating group-adaptive elastic net penalties.
method Derive low-dimensional representation of Taylor approximation for marginal likelihood and its derivative for group-adaptive ridge penalties; approximate elastic net marginal likelihood by ridge; transform ridge penalties to elastic net penalties.
result Significantly decreases computation time and outperforms other methods.
Variational Prediction simplifies Bayesian inference without test time costs.
problem Bayesian inference's computational costs and posterior predictive distribution marginalization.
method Variational Prediction learns a variational approximation to the posterior predictive distribution using a variational bound.
result Directly learns a variational approximation to the posterior predictive distribution without test time marginalization costs.
Proposes MFSWB for marginal fairness in SWB, improving efficiency and performance.
problem Achieving marginal fairness in SWB averaging.
method Defining MFSWB as a constrained SWB problem, proposing two surrogate problems and a new slicing distribution.
result Surrogate MFSWB problems effectively minimize distances to marginals and encourage marginal fairness.
This manuscript shows that AdaBoost and its immediate variants can produce approximate maximum margin classifiers simply by scaling step size choices with a fixed small constant. In this way, when the unscaled step size is an optimal choice, these results provide guarantees for Friedman's empirically successful "shrink…