This work extends alpha-beta divergences to complex data and finds closed-form solutions.
problem Approximating complex random vectors.
method Extending alpha-beta divergences to complex data and optimizing the alpha-beta mean distortion.
result Closed-form expression for the centroid of complex random vectors.
PED-ANOVA efficiently calculates HP importance in arbitrary subspaces.
problem Understanding the role of different hyperparameters in arbitrary subspaces.
method Derive a novel f-ANOVA formulation for arbitrary subspaces and use Pearson divergence (PED) for a closed-form calculation of HP importance.
result Demonstrates successful identification of important HPs in different subspaces.
The paper analyzes SBL pruning criteria under weakened assumptions.
problem Sparse Bayesian learning hyperparameter divergence and pruning.
method Analyzing marginal likelihood function under weakened Gaussian assumptions.
result Conditions for finite vs infinite hyperparameters lead to F-SBL pruning.
A new optimisation framework for neural networks without hyperparameters.
problem Lack of explicit architectural information in optimisation frameworks.
method Extends mirror descent to account for neural architecture, transforming a Bregman divergence.
result Automatic gradient descent: a first-order optimiser without hyperparameters.
Proposes practical kernel tests for f-divergences with theoretical guarantees.
problem Two-sample testing and machine unlearning evaluation.
method Regularized f-divergence kernel tests, adaptive to hyperparameters. result Different f-divergences highlight localized differences. A study on α-GANs proving convergence and estimation guarantees.
problem Analyzing the convergence and estimation guarantees of α-GANs. method Proved a correspondence between α-GANs and f-divergences, and provided estimation bounds. result Estimation bounds indicate diverse GAN behavior as a function of α. New method uses KL-divergence to create non-informative priors for multivariate Gaussian.
problem Handling hyperparameters for non-informative limits in multivariate Gaussian conjugate priors.
method Using scaled KL-divergence between multivariate Gaussians to construct Wishart and normal-Wishart conjugate priors.
result Forming non-informative priors without violating Wishart shape parameter restrictions.
The study examines how hyperparameters affect prediction discrepancies in machine learning models.
problem Prediction inconsistencies across different machine learning models trained on the same dataset.
method Investigation of six models (Elastic Net, Decision Tree, k-NN, SVM, RF, XGBoost) on 21 benchmark datasets, focusing on key hyperparameters.
result Hyperparameter tuning improves model performance but increases prediction discrepancies, especially in Extreme Gradient Boosting.
The paper introduces a new divergence measure for variational autoencoders to improve reconstruction and generation.
problem Balancing reconstruction and generalizability in latent space of variational autoencoders.
method Presented a regularisation mechanism based on skew-geometric Jensen-Shannon divergence.
result The skew-geometric Jensen-Shannon divergence leads to better reconstruction and generation in variational autoencoders.
Proposes a guaranteed regularization method for maximum likelihood estimation using gauge symmetry in Kullback-Leibler divergence.
problem Overfitting in maximum likelihood estimation.
method Introduces a regularization approach based on gauge symmetry in Kullback-Leibler divergence.
result The method provides a theoretically guaranteed optimal model without frequent hyperparameter tuning.
SMT trains generative models by estimating mixture scores, outperforming existing methods.
problem Training one-step generative models efficiently and effectively.
method Score-of-Mixture Training (SMT) estimates the score of mixture distributions between real and fake samples.
result SMT/SMD outperform existing methods on CIFAR-10 and ImageNet 64x64 datasets.
New methods stabilize EEG classification performance across subjects.
problem Large performance drop in EEG classification models on unseen subjects.
method Regularization techniques using divergence estimation.
result Significant increase in balanced accuracy on test subjects.
This research provides theoretical guarantees for hyperparameter estimation in complex network dynamical systems.
problem Theoretical guarantees for hyperparameter estimation in large, inhomogeneous complex network dynamical systems.
method Formulating the system's evolution in a measure transport perspective, proposing a theoretical framework for estimating hyperparameters with mean-type observations.
result A nonasymptotic bound for the deviation of hyperparameter estimates in inhomogeneous complex network dynamical systems with respect to network population size.
A new method Expectigrad improves on Adam and RMSProp by reducing divergence and improving performance.
problem Improving the convergence properties of adaptive gradient methods like Adam and RMSProp.
method Adjusts stepsizes using a per-component unweighted mean of all historical gradients and a bias-corrected momentum term.
result Cannot diverge on convex optimization problems that cause Adam to diverge.
Meta-learning improves performance across similar tasks in adversarial bandit settings.
problem Improving performance across multiple similar tasks in adversarial bandit scenarios.
method Designing meta-algorithms that combine outer learners to tune hyperparameters of inner learners for MAB and BLO.
result Meta-algorithms improve task-averaged regret for MAB and BLO, showing direct relationship with action space-dependent measures.
BaM improves BBVI by optimizing a score-based divergence, leading to faster convergence.
problem Slow convergence of black-box variational inference methods.
method Batch and match (BaM) approach based on a score-based divergence.
result BaM converges exponentially quickly to the target mean and covariance.
Researchers establish bounds for SGMs' KL and Wasserstein divergences under various noise schedules.
problem Estimating the error between target and estimated distributions in SGMs.
method Established upper bounds for KL divergence and Wasserstein distance, incorporating target distribution properties and SGM hyperparameters.
result Optimal noise schedules identified for SGMs, improving generative quality.
By exploiting the property that the RBM log-likelihood function is the difference of convex functions, we formulate a stochastic variant of the difference of convex functions (DC) programming to minimize the negative log-likelihood. Interestingly, the traditional contrastive divergence algorithm is a special case of th…
EGAB algorithms improve online portfolio selection.
problem Online portfolio selection problem.
method Generalized exponentiated gradient (EG) updates with Alpha-Beta divergence regularization.
result EGAB algorithms enhance portfolio performance, especially with transaction costs.
New GPs model edge functions on complex networks, capturing divergence and curl.
problem Modeling flow data on networks with independent learning of Hodge components.
method Developed Hodge-compositional edge GPs using Hodge decomposition.
result Hodge-compositional edge GPs can represent any edge function and capture flow relevance.
New MD algorithms using Tempesta logarithms for machine learning.
problem Optimization in machine learning with tailored hyperparameters.
method Developed Mirror Descent algorithms using Tempesta multi-parametric logarithms.
result Wide and flexible family of Mirror Descent and mirror-less updates.
A new algorithm reduces bias and variance in distributionally robust optimization.
problem Distributionally robust optimization with bias and variance issues.
method Prospect, a stochastic gradient-based algorithm that reduces hyperparameter tuning.
result Prospect achieves linear convergence and 2-3x faster convergence on various benchmarks.
Single training run learns optimal VAE parameters for various β values.
problem Training VAEs with varying β values for optimal trade-off between distortion and rate.
method Introduced Multi-Rate VAE (MR-VAE) using hypernetworks to map β to optimal parameters.
result MR-VAEs can construct the full rate-distortion curve without additional training.
Enhanced DeepONet framework with uncertainty quantification for complex operators.
problem Learning complex operators with uncertainty quantification.
method Generalised variational inference (GVI) using Rényi's α-divergence.
result Superior predictive accuracy and uncertainty quantification.
Study optimizes tree-based models for better alignment of predicted scores and actual probabilities.
problem Traditional calibration metrics fail to align predicted scores with actual probabilities when score distributions deviate from the underlying data.
method Optimizes tree-based models (Random Forest, XGBoost) using Kullback-Leibler (KL) divergence to minimize the difference between predicted and true probability distributions.
result Optimized tree-based models yield superior alignment between predicted scores and actual probabilities without significant performance loss.
New method improves online nonparametric estimators with minimal extra computation.
problem Model selection and hyperparameter tuning for online nonparametric estimators.
method Weighted rolling validation procedure for online cross-validation.
result Improves base estimators to achieve better heuristic performance and adaptive convergence rate.
Pre-trained GPs improve Bayesian optimization efficiency.
problem Optimizing complex machine learning models requires domain knowledge but is challenging.
method Developed HyperBO framework using pre-trained Gaussian processes.
result HyperBO achieves up to 3x better efficiency in hyperparameter tuning.
Non-negative matrix factorization (NMF) is a knowledge discovery method that is used in many fields. Variational inference and Gibbs sampling methods for it are also wellknown. However, the variational approximation error has not been clarified yet, because NMF is not statistically regular and the prior distribution us…
A new method stabilizes deep reinforcement learning by using QGraphs to retain replay memory information.
problem Stabilizing model-free off-policy deep reinforcement learning with soft divergence.
method Representing past experiences as a QGraph, selecting a subgraph with favorable structure, and using lower bounds for temporal difference learning.
result QG-DDPG method is less prone to soft divergence and more robust to hyperparameters.
The Poisson model is frequently employed to describe count data, but in a Bayesian context it leads to an analytically intractable posterior probability distribution. In this work, we analyze a variational Gaussian approximation to the posterior distribution arising from the Poisson model with a Gaussian prior. This is…
t-SNE fails to reveal clusters even in well-clusterable data.
problem t-SNE's failure to show clusters in well-clusterable data.
method Empirical evidence and theoretical analysis of t-SNE's limitations.
result t-SNE may hide clustering patterns even with strong signals in the data.
Proposes a new approach to regression learning that addresses overfitting and underfitting.
problem Regression learning issues, including overfitting and underfitting.
method Introduces epsilon-Confidence Approximately Correct (epsilon CoAC) framework using Kullback Leibler divergence.
result Demonstrates improved learnability and accuracy compared to cross-validation.
The burgeoning success of deep learning has raised the security and privacy concerns as more and more tasks are accompanied with sensitive data. Adversarial attacks in deep learning have emerged as one of the dominating security threat to a range of mission-critical deep learning systems and applications. This paper ta…
GCVAE improves disentanglement in VAEs while balancing reconstruction error.
problem Improving disentanglement in VAEs while maintaining low reconstruction error.
method Introduces three controllable Lagrangian hyperparameters to optimize reconstruction and KL divergence loss.
result GCVAE outperforms state-of-the-art models in disentanglement while balancing reconstruction.
We identity a by-far-unrecognized problem of Adam-style optimizers which results from unnecessary coupling between momentum and adaptivity. The coupling leads to instability and divergence when the momentum and adaptivity parameters are mismatched. In this work, we propose a method, Laprop, which decouples momentum and…
ControlVAE improves VAE performance by adding a controller to tune hyperparameters.
problem Existing VAE models struggle with KL vanishing and low reconstruction quality.
method ControlVAE combines a controller inspired by automatic control theory with VAE to improve performance.
result ControlVAE achieves better disentangling and reconstruction quality than existing methods.
Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters. We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters …
New diffusion models capture heavy-tailed distributions better.
problem Diffusion models struggle with rare or extreme events in heavy-tailed distributions.
method Repurposed diffusion framework using multivariate Student-t distributions, tailored perturbation kernel, and γ-divergence. result Our models generate rare and extreme events more effectively than standard diffusion models.
Extends hyperparameter transfer across model sizes and modules, improving training speed.
problem Training stability and performance of large-scale models with optimal hyperparameters.
method Complete(d) Parameterisation, per-module hyperparameter optimisation and transfer. result Hyperparameter transfer holds even in the per-module hyperparameter regime, improving training speed.
HASSO improves SO algorithms by dynamically tuning hyperparameters.
problem Inefficiency of hyperparameter tuning for SO algorithms.
method HASSO is a self-adjusting SO algorithm that dynamically tunes its own hyperparameters.
result HASSO enhances the performance of various SO algorithms across different test problems.
Study shows leaving some hyperparameters at default can outperform tuning them.
problem Importance of tuning hyperparameters in machine learning performance.
method Non-inferiority test and tuning risk methodology.
result Leaving some hyperparameters at default can outperform tuning them.
Deep learning models are full of hyperparameters, which are set manually before the learning process can start. To find the best configuration for these hyperparameters in such a high dimensional space, with time-consuming and expensive model training / validation, is not a trivial challenge. Bayesian optimization is a…
New method for tuning Graphical Lasso hyperparameters.
problem Tuning hyperparameters of Graphical Lasso.
method Bilevel optimization with first-order method.
result Derivation of Graphical Lasso Jacobian.
Hyperparameter tuning is an omnipresent problem in machine learning as it is an integral aspect of obtaining the state-of-the-art performance for any model. Most often, hyperparameters are optimized just by training a model on a grid of possible hyperparameter values and taking the one that performs best on a validatio…
Hyperparameters are critical in machine learning, as different hyperparameters often result in models with significantly different performance. Hyperparameters may be deemed confidential because of their commercial value and the confidentiality of the proprietary algorithms that the learner uses to learn them. In this …
New optimizer improves privacy-protected hyperparameter tuning.
problem No practical methods for differentially private hyperparameter selection.
method Study honest hyperparameter selection under DP, show adaptive optimizers like DPAdam have an advantage.
result DPAdam optimizes hyperparameters more efficiently under DP constraints.
Paper defines hyperparameter importance for efficient tuning.
problem Computational inefficiency in tuning all hyperparameters.
method Defines hyperparameter importance via subsampling procedures.
result Proposed importance consistent with full data under weak conditions.
WRS improves CNN hyperparameter optimization.
problem Finding optimal hyperparameters for CNNs efficiently.
method Combination of Random Search and probabilistic greedy heuristic.
result WRS outperforms other methods in CNN hyperparameter optimization.