Large models collapse epistemic uncertainty, challenging traditional wisdom.
problem Epistemic uncertainty collapse in large models.
method Implicit ensembling and decomposition techniques.
result Larger models can collapse epistemic uncertainty, contrary to expectations.
Paper proposes a new method for probabilistic electricity price forecasting.
problem Accurate estimation of forecast uncertainties for optimal decision making.
method Implicit generative ensemble post-processing using an ensemble of point forecasting models.
result Method outperforms well-established model combination benchmarks.
EnCF improves data assimilation for implicit, non-smooth observations.
problem Data assimilation challenges with implicit, many-to-one observations.
method EnCF uses a stochastic controlled flow to update forecast distributions.
result EnCF outperforms Kalman filters for non-Gaussian, implicit observations.
Paper explores how DPP sampling can implicitly regularize kernel regression.
problem Improving kernel regression by reducing redundancy in data.
method Using Determinantal Point Processes (DPPs) to sample subsets implicitly regularizes ridgeless Kernel Regression.
result Ensemble of ridgeless regressors can be effective for datasets with redundant information.
This work improves distribution recovery from sparse data using Random Forest implicit regularization.
problem Distribution recovery from limited statistics.
method Closed-form estimator for scaled beta distributions, using composite quantile and moment matching.
result Improved classification accuracy through closed-form distribution recovery and implicit regularization.
Ensemble methods that average over a collection of independent predictors that are each limited to a subsampling of both the examples and features of the training data command a significant presence in machine learning, such as the ever-popular random forest, yet the nature of the subsampling effect, particularly of th…
Creating maps is an essential task in robotics and provides the basis for effective planning and navigation. In this paper, we learn a compact and continuous implicit surface map of an environment from a stream of range data with known poses. For this, we create and incrementally adjust an ensemble of approximate Gauss…
Residual networks' depth is mathematically equivalent to expanding an implicit ensemble size.
problem Understanding why deep residual networks are effective.
method Formal analysis of residual networks as ensembles of shallow models.
result Increasing network depth is equivalent to expanding the size of an implicit ensemble, revealing a hierarchical structure.
Enhances optimization and sampling methods using ensemble-based gradient inference.
problem Improving ensemble-based methods for optimization and sampling.
method Ensemble-based gradient inference (EGI) to extract higher-order derivatives from particle ensembles.
result Augmented algorithms outperform gradient-free variants, especially in multimodal and non-Gaussian settings.
XEM improves multivariate time series classification with explainable models.
problem Multivariate time series classification challenges.
method Hybrid ensemble method combining explicit boosting-bagging and implicit divide-and-conquer.
result XEM outperforms state-of-the-art MTS classifiers on public datasets.
We explore an ensembled Σ-net for fast parallel MR imaging, including parallel coil networks, which perform implicit coil weighting, and sensitivity networks, involving explicit sensitivity maps. The networks in Σ-net are trained in a supervised way, including content and GAN losses, and with various ways of data c…
JoVA combines two VAEs to learn user and item representations for better recommendation.
problem Collaborative filtering with implicit feedback.
method Joint Variational Autoencoders (JoVA) with a hinge-based pairwise loss function (JoVA-Hinge).
result JoVA-Hinge outperforms state-of-the-art methods in top-k recommendation.
Tree ensembles like RF and GBT can be seen as kernels, improving regression and classification performance.
problem Improving kernel methods for tree ensemble based models.
method Investigation of RF and GBT kernels in simulation and real data.
result RF and GBT kernels are competitive to their respective ensembles in higher dimensions, particularly with noisy features.
We improve prediction risk estimation for large datasets using sketching and ridge regression.
problem Estimating prediction risks for large datasets efficiently and accurately.
method Random matrix theory, generalized cross validation, sketched ridge regression ensembles, and ensemble trick.
result Consistent risk estimation and prediction intervals for large-scale datasets.
Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic r…
The dynamic ensemble selection of classifiers is an effective approach for processing label-imbalanced data classifications. However, such a technique is prone to overfitting, owing to the lack of regularization methods and the dependence of the aforementioned technique on local geometry. In this study, focusing on bin…
Study shows how feature weighting affects neural network regularization.
problem Understanding how feature weighting influences neural network regularization.
method Derived equivalence paths connecting different weighting matrices and ridge regularization levels.
result Ridge estimators trained on weighted features are asymptotically equivalent when evaluated against test vectors.
Stochastic mirror descent improves performance on ensemble models.
problem Improving performance of ensemble models using stochastic mirror descent.
method Utilizes mirror potential to influence training algorithm's implicit bias, mapping evolution to continuous time process.
result Converges to a nonlinear PDE in asymptotic regime of large networks, with mirror potential affecting gradient flow.
The paper analyzes the risk of bagging regularized M-estimators under proportional asymptotics.
problem Characterizing the risk of ensemble estimators trained with subsamples and regularizers.
method Developed a consistent estimator for the risk of ensemble estimators under proportional asymptotics.
result Optimal subsample size k⋆ tends to be in the overparameterized regime for the full-ensemble estimator. RL struggles with generalization due to implicit partial observability.
problem Generalization in RL is difficult due to implicit partial observability.
method Re-cast RL problem as solving epistemic POMDPs and propose ensemble-based techniques.
result Simple ensemble-based technique achieves significant generalization gains.
RealStats detects fake images rigorously, combining multiple detectors for robustness.
problem Detecting AI-generated images remains challenging due to evolving generative models.
method Combines training-free statistics to compute p-values and aggregate them for a unified real-image distribution.
result Framework produces interpretable probability scores for robust fake image detection.
This paper describes the system developed at Amobee for the WASSA 2018 implicit emotions shared task (IEST). The goal of this task was to predict the emotion expressed by missing words in tweets without an explicit mention of those words. We developed an ensemble system consisting of language models together with LSTM-…
Bagging stabilizes linear interpolators, improving their generalization performance.
problem Unstable linear interpolators fail on noisy data.
method Introduced multiplier-bootstrap-based bagged least square estimator.
result Bagging effectively mitigates variance, leading to bounded prediction risk.
The paper addresses model averaging and ensembling, providing theoretical and practical insights.
problem Combining least squares estimators from multiple candidate models for improved predictive accuracy.
method Establishes oracle inequalities for Mallows' Cp criterion, proposes a novel Mallows-type MA procedure. result Demonstrates the effectiveness of the proposed Mallows-type MA estimator through numerical experiments.
This research uses DPPs to improve semi-parametric regression models.
problem Improving comprehensibility in semi-parametric regression models without sacrificing accuracy.
method Introduced a novel representation of finite DPPs and used it to derive a key identity illustrating implicit regularization.
result Demonstrated the implicit regularization effect of determinantal sampling for semi-parametric regression.
We introduce a new deep convolutional neural network, CrescendoNet, by stacking simple building blocks without residual connections. Each Crescendo block contains independent convolution paths with increased depths. The numbers of convolution layers and parameters are only increased linearly in Crescendo blocks. In exp…
The recently proposed self-ensembling methods have achieved promising results in deep semi-supervised learning, which penalize inconsistent predictions of unlabeled data under different perturbations. However, they only consider adding perturbations to each single data point, while ignoring the connections between data…
Better uncertainty estimates for neural networks using Gaussian process priors.
problem Poor uncertainty estimates in neural networks, especially on out-of-distribution data.
method Characterize the function-space prior of an ensemble of infinitely-wide neural networks as a Gaussian process and use it to build a probabilistic model.
result The approach improves calibration of neural networks, especially under distributional shift.
Generative Bayesian Computation improves surrogates for expensive simulations.
problem Limitations of Gaussian process surrogates in handling complex, non-stationary data.
method Generative Bayesian Computation via Implicit Quantile Networks (IQNs).
result Generative Bayesian Computation outperforms traditional Gaussian process methods across various benchmarks.
This paper shows how to train only the implicit layer of overparameterized implicit neural networks.
problem Understanding how the implicit layer contributes to the training of overparameterized implicit neural networks.
method Restricting training to only the implicit layer and analyzing the generalization error for ReLU-activated networks.
result Global convergence is guaranteed even if only the implicit layer is trained, and gradient flow with proper random initialization can achieve small generalization errors.
Paper studies the theoretical equivalence between implicit and explicit neural networks in high dimensions.
problem Lack of theoretical analysis of implicit and explicit neural networks.
method Examined high-dimensional implicit neural networks and established their equivalence to explicit networks.
result Equivalence between implicit and explicit neural networks in high dimensions.
Gradient matching method estimates implicit regularization in complex deep learning systems.
problem Estimating implicit regularization in modern deep learning systems with complex modifications.
method Gradient matching methods to empirically estimate implicit regularization.
result Empirical estimation of implicit regularization in arbitrary networks, including dropout.
Wasserstein Dropout improves uncertainty estimation in neural networks.
problem Estimating neural uncertainties for safe machine learning.
method A purely non-parametric approach using dropout-based sub-network distributions and Wasserstein distance.
result Wasserstein Dropout outperforms state-of-the-art methods in uncertainty estimation.
Continuous semi-implicit models enable faster training and better performance in generative modeling.
problem Slow convergence in hierarchical semi-implicit models during training.
method CoSIM, a continuous semi-implicit model that incorporates a continuous transition kernel for efficient training.
result CoSIM achieves superior performance on image generation tasks compared to existing methods.
Gradient descent converges to a global minimum in nonlinear ReLU implicit networks with linear width.
problem Understanding convergence of gradient methods in nonlinear, infinitely deep ReLU networks.
method Introduced a scaling constant to ensure well-posedness of the equilibrium equation, proving convergence to a global minimum for linear width networks.
result Gradient descent converges to a global minimum at a linear rate for nonlinear ReLU implicit networks with linear width.
This work improves neural network calibration using explicit regularization.
problem Improving predictive uncertainty in neural networks.
method Introducing a probabilistic calibration measure and exploring explicit regularization techniques.
result Explicit regularization improves log-likelihood and predictive uncertainty.
Study finds implicit government guarantee improves municipal investment bond ratings.
problem Questioning the objectivity of municipal investment bond ratings due to implicit government guarantee.
method Text mining of policy documents and PMC index model for implicit guarantee strength calculation.
result Implicit government guarantee boosts municipal investment bond ratings, especially in less developed regions.
Study shows SGD's generalization is not explained by implicit bias.
problem Explaining the generalization ability of overparameterized learning algorithms.
method Revisited Stochastic Convex Optimization with SGD, demonstrating limitations of implicit bias.
result No distribution-independent or distribution-dependent implicit regularizer can explain SGD's generalization.
This paper measures the intensity of implicit government guarantees using PMC index model.
problem Excessive local government debt due to implicit government guarantees.
method Text mining of policy documents related to municipal investment bonds, PMC index model.
result Recent policies have reduced the intensity of implicit government guarantees.
The paper explains implicit regularization in hierarchical tensor factorization and deep CNNs.
problem Understanding implicit regularization in complex neural network architectures.
method Theoretical analysis using dynamical systems to overcome challenges in hierarchy.
result Established implicit regularization towards low hierarchical tensor rank, equivalent to locality in CNNs.
In this paper, we describe the "implicit autoencoder" (IAE), a generative autoencoder in which both the generative path and the recognition path are parametrized by implicit distributions. We use two generative adversarial networks to define the reconstruction and the regularization cost functions of the implicit autoe…
Implicit discourse relation classification is of great challenge due to the lack of connectives as strong linguistic cues, which motivates the use of annotated implicit connectives to improve the recognition. We propose a feature imitation framework in which an implicit relation network is driven to learn from another …
ETGPSSM efficiently models high-dimensional, non-stationary systems with reduced complexity.
problem Prohibitive computational and parametric complexity in high-dimensional, non-stationary dynamical systems.
method ETGPSSM integrates a single shared GP with input-dependent normalizing flows for scalable and flexible modeling.
result ETGPSSM outperforms existing models in computational efficiency and accuracy.
Deep tensor factorization benefits from implicit regularization with polynomial growth.
problem Tensor factorization's implicit regularization effect in deep networks is not well understood.
method Investigated the implicit regularization in deep tensor factorization, showing polynomial growth.
result Implicit regularization in deep tensor factorization grows polynomially with depth, improving estimation accuracy and convergence.
Gradient descent on ReLU networks with square loss implicitly favors balanced weights.
problem Understanding implicit regularization in nonlinear neural networks with regression losses.
method Analyzing gradient descent dynamics on ReLU networks with square loss.
result It is impossible to characterize the implicit regularization of ReLU networks with square loss by any explicit function of model parameters.
We extend the existing framework of semi-implicit variational inference (SIVI) and introduce doubly semi-implicit variational inference (DSIVI), a way to perform variational inference and learning when both the approximate posterior and the prior distribution are semi-implicit. In other words, DSIVI performs inference …
Deep equilibrium models converge globally without explicit computation.
problem Global convergence of deep learning models with implicit layers.
method Analysis of gradient dynamics and proof of convergence rate.
result Deep equilibrium models converge to global optimum at a linear rate.
Proposes efficient, modular method for implicit differentiation.
problem Implicit differentiation of optimization problems.
method Automatic implicit differentiation using autodiff and implicit function theorem.
result Automatic differentiation of optimization problems is made easier and more modular.