Introduces new gradient-based methods for machine learning problems.
problem New challenges in machine learning due to decision-making and multi-agent problems.
method Gradient-based optimization and variational inequalities.
result Shifts focus from pattern recognition to decision-making and multi-agent problems.
Gradient-based MCMC for discrete spaces improves sampling performance.
problem Sampling in discrete spaces using traditional methods is challenging.
method Introduced new discrete Metropolis-Hastings samplers inspired by MALA, with a novel preconditioning technique.
result Demonstrated strong empirical performance across various challenging sampling problems.
We formulate a general framework for competitive gradient-based learning that encompasses a wide breadth of multi-agent learning algorithms, and analyze the limiting behavior of competitive gradient-based learning algorithms using dynamical systems theory. For both general-sum and potential games, we characterize a non…
Meta learning works well with overparameterized models, a phenomenon called 'benign overfitting'.
problem Understanding why overparameterized models perform well in few-shot learning.
method Analyzed the generalization performance of gradient-based meta learning with an overparameterized meta linear regression model.
result Demonstrated that overparameterized meta learning can still generalize well, a phenomenon called 'benign overfitting'.
Gradient-based meta-learners such as Model-Agnostic Meta-Learning (MAML) have shown strong few-shot performance in supervised and reinforcement learning settings. However, specifically in the case of meta-reinforcement learning (meta-RL), we can show that gradient-based meta-learners are sensitive to task distributions…
In this work we study generalization of neural networks in gradient-based meta-learning by analyzing various properties of the objective landscapes. We experimentally demonstrate that as meta-training progresses, the meta-test solutions, obtained after adapting the meta-train solution of the model, to new tasks via few…
Causal structure learning has been a challenging task in the past decades and several mainstream approaches such as constraint- and score-based methods have been studied with theoretical guarantees. Recently, a new approach has transformed the combinatorial structure learning problem into a continuous one and then solv…
Gradient-based methods improve understanding of deep learning survival models.
problem Limited interpretability of deep learning survival models hinders their adoption.
method Gradient-based explanation methods tailored to survival neural networks.
result Gradient-based methods capture feature effects and temporal dynamics.
Generative model initializes 2-layer network weights for small datasets.
problem Approximating functions with 2-layer networks using small datasets and gradient-based training.
method Initialize hidden weights with a learned proposal distribution parameterized as a deep generative model. Refine with gradient-based post-processing and regularization.
result Demonstrates effectiveness of the approach with numerical examples.
Combines gradient-based and competitive learning for unsupervised feature extraction.
problem Handling input data without supervision and replicating input manifold topology.
method Integrates gradient-based and competitive learning approaches to learn topological structures.
result The dual competitive layer outperforms the vanilla layer in high-dimensional datasets.
This paper surveys gradient-based multi-objective deep learning methods.
problem Balancing multiple conflicting objectives in deep learning models.
method Gradient-based techniques adapted from Multi-Objective Optimization.
result Comprehensive survey of gradient-based multi-objective deep learning algorithms.
Proposes VSGD optimizer combining probabilistic and gradient-based methods.
problem Uncertainty modeling in deep neural networks.
method Combines probabilistic and gradient-based approaches using SVI.
result VSGD outperforms Adam and SGD on image classification tasks.
We propose a technique for increasing the efficiency of gradient-based inference and learning in Bayesian networks with multiple layers of continuous latent vari- ables. We show that, in many cases, it is possible to express such models in an auxiliary form, where continuous latent variables are conditionally determini…
MPLP learns neural network weights by treating operations as message-passing agents.
problem Training neural networks using gradient-based methods.
method MPLP abstracts neural network operations as message-passing agents, updating internal states and passing messages.
result MPLP outperforms traditional gradient-based methods on simple feed-forward neural networks.
New method combines QQA and gradient-based sampling for combinatorial optimization.
problem Scalability challenges in learning-based solvers for combinatorial optimization.
method Integrates gradient-based update through continuous relaxation with Quasi-Quantum Annealing (QQA) and parallel communication.
result Achieves superior speed-quality trade-offs for large-scale instances.
Proposes an automatic cyclical scheduling for gradient-based discrete sampling.
problem Gradient-based sampling in high-dimensional models can get stuck in local modes.
method Cyclical step size and balancing schedules with automatic hyperparameter tuning.
result Proves non-asymptotic convergence and inference guarantees for general discrete distributions.
Gradient-based methods can be biased by distributional asymmetries in bivariate categorical data.
problem Gradient-based causal discovery methods can be biased by distributional asymmetries in bivariate categorical data.
method Identified and examined two distributional biases: Marginal Distribution Asymmetry and Marginal Distribution Shift Asymmetry. Employed two simple models to demonstrate and control these biases.
result Gradient-based methods can be biased by distributional asymmetries, and these biases can be controlled.
Novel BSG method for efficient stochastic optimization.
problem Efficient optimization of non-convex surfaces in stochastic settings.
method Binary search combined with first order gradient optimization.
result BSG produces more promising results and better generalization than other methods.
New method creates universal perturbations to fool neural network interpretations.
problem Vulnerability of gradient-based saliency maps to adversarial perturbations.
method Gradient-based optimization and PCA-based approach to create UPI.
result Existence and successful application of Universal Perturbation for Interpretation (UPI).
Paper presents a novel gradient-based method for training models and hyperparameters simultaneously.
problem Achieving generalization in machine learning models.
method A novel gradient-based framework that trains parameters and hyperparameters simultaneously.
result Significantly smaller runtime compared to benchmark methods for equivalent prediction scores.
Paper analyzes dataset distillation for efficient encoding of task-relevant information.
problem Efficiently encoding task-relevant information from gradient-based learning of non-linear tasks.
method Theoretical analysis of dataset distillation applied to two-layer neural networks with gradient-based training.
result Low-dimensional structure of the problem is efficiently encoded into distilled data, reproducing a model with high generalization ability.
Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these …
Method generates time-series attribution maps with identifiability guarantees.
problem Lack of identifiability guarantees in gradient-based attribution methods.
method Regularized contrastive learning algorithm trained on time-series data.
result Empirically shows robust approximation of zero vs. non-zero entries in the ground-truth attribution map.
New method improves combinatorial optimization by overcoming inefficient sampling.
problem Wandering in contours: sampling similar solutions for long periods.
method Reheated Gradient-based Discrete Sampling with a novel mechanism.
result Superior performance across various combinatorial optimization problems.
GBML with deep nets converges globally and generalizes well.
problem Theoretical guarantees for few-shot learning with deep nets.
method Proving global convergence and generalization bounds for GBML with over-parameterized DNNs.
result GBML with over-parameterized DNNs converges globally to the optimum at a linear rate and achieves good generalization.
New method preserves convergence rates in gradient-based optimization.
problem How to discretize gradient-based optimization systems while preserving stability and convergence rates.
method Geometric framework for dissipative symplectic integration.
result Dissipative symplectic integrators preserve rates of convergence up to a controlled error.
In modern data analysis, random sampling is an efficient and widely-used strategy to overcome the computational difficulties brought by large sample size. In previous studies, researchers conducted random sampling which is according to the input data but independent on the response variable, however the response variab…
Gradient-based methods introduce noise that penalizes models sensitive to weight perturbations.
problem Noise in gradient-based optimization methods.
method Analysis of Gradient Descent (GD) and Stochastic Gradient Descent (SGD) updating all parameters simultaneously.
result Noise introduced by simultaneous parameter updates penalizes models sensitive to weight perturbations.
New bounds show faster convergence for learning algorithms.
problem Improving risk bounds for learning algorithms.
method Using algorithmic stability and common assumptions like Polyak-Lojasiewicz condition, smoothness, and Lipschitz continuity.
result Achieves convergence rate of O(log2(n)/n2) with high probability. Vulnerability to adversarial attacks is one of the principal hurdles to the adoption of deep learning in safety-critical applications. Despite significant efforts, both practical and theoretical, the problem remains open. In this paper, we analyse the geometry of adversarial attacks in the large-data, overparametrized …
A gradient-based method learns the structure of TAN for Bayesian network classifiers.
problem Learning the structure of Bayesian networks is difficult.
method A distribution over graph structures learned via gradient-based optimization.
result Consistently outperforms random and Chow-Liu TAN structures.
GIT uses gradient estimators to target interventions for causal discovery.
problem Challenges in inferring causal structure from observational data.
method GIT uses gradient estimators to target interventions for causal discovery.
result GIT performs on par with competitive baselines, surpassing them in low-data regimes.
Extends gradient-based optimization to spline functions.
problem Limitations of standard differentiable programming methods.
method Derives Jacobian of spline functions and uses it in predictive models.
result Improved performance in various applications.
Background: It is still an open research area to theoretically understand why Deep Neural Networks (DNNs)---equipped with many more parameters than training data and trained by (stochastic) gradient-based methods---often achieve remarkably low generalization error. Contribution: We study DNN training by Fourier analysi…
Fully automating machine learning pipelines is one of the key challenges of current artificial intelligence research, since practical machine learning often requires costly and time-consuming human-powered processes such as model design, algorithm development, and hyperparameter tuning. In this paper, we verify that au…
This work investigates how gradient-based learning performs with structured data, revealing issues and improvements.
problem Gradient-based learning under structured data, particularly with a spiked covariance structure.
method Investigates the effect of a spiked covariance structure on gradient-based feature learning and proposes weight normalization.
result Gradient-based dynamics may fail to recover the true direction in anisotropic settings, but weight normalization can improve performance.
A novel gradient-based method optimizes decision trees for complex tasks.
problem Training decision trees with arbitrary differentiable loss functions.
method Gradient-based optimization using first and second derivatives of loss functions.
result Improves accuracy and flexibility in decision tree optimization.
Randomized gradient-based ensemble improves prediction accuracy.
problem Improving prediction accuracy in machine learning.
method Randomization and gradient-based aggregation of weakly-correlated estimators.
result The method outperforms existing techniques in terms of increased accuracy.
We consider an online learning process to forecast a sequence of outcomes for nonconvex models. A typical measure to evaluate online learning algorithms is regret but such standard definition of regret is intractable for nonconvex models even in offline settings. Hence, gradient based definition of regrets are common f…
Gradient-based training and pruning for radial basis function networks in materials physics.
problem Interpretable and robust machine learning for materials physics problems.
method Gradient-based training and pruning of radial basis function networks with closed-form optimization criteria.
result Pruned models provide compact and interpretable versions of larger models, offering insights into atom-level migration processes.
Charmer improves character-level adversarial attacks for language models.
problem Efficiency and effectiveness of character-level adversarial attacks for language models.
method Query-based adversarial attack method that maintains semantic similarity.
result Charmer achieves high attack success rate and similar adversarial examples.
Generalization in deep neural networks can be analyzed using minimax rates for gradient methods.
problem Generalization performance of over-parameterized neural networks
method Establishing a connection between gradient-based methods and kernel methods
result Deriving minimax-optimal rates for GD and SGD under polynomial network width scaling
GFM models neural network training as a dynamical system to forecast final weights.
problem Computational intensity and inefficiency in training deep neural networks.
method Gradient Flow Matching (GFM) treats training as a dynamical system with learned vector fields.
result GFM achieves forecasting accuracy competitive with Transformer-based models and significantly outperforms classical baselines.
TrIM improves gradient-based dimension reduction and regression.
problem Efficiently identifying relevant feature subspace for high-dimensional regression.
method Introduced TrIM forest, an iterative approach using Mondrian forest and EGOP estimate.
result Consistency guarantees and convergence rates for EGOP matrix and random forest estimator.
The paper proposes a gradient-based method for multi-penalty Ridge regression.
problem Optimizing multiple regularization hyperparameters for linear regression.
method Gradient-based optimization through matrix differential calculus.
result The method outperforms traditional regularization techniques like LASSO and Ridge.
Method generates visual explanations for similarity models without classification.
problem Lack of visual explanations for similarity models trained without classification loss.
method Gradient-based visual attention using learned feature embeddings.
result Attention maps improve model performance and can be used as constraints.
Adaptor 'E' extends gradient-based optimizers to explore loss landscapes, improving generalization.
problem Finding lower and better-generalizing minima in deep learning.
method Proposes an adaptor 'E' to extend gradient-based optimizers, encouraging exploration along landscape valleys.
result Adapted optimizers increase test accuracy by an average of 2.5% in large-batch training tasks.
Hybrid method improves sampling from multimodal distributions.
problem Sampling from multimodal posterior distributions efficiently.
method Jump-Diffusion Langevin Dynamics hybrid with Metropolis.
result Calibrated hybrid method outperforms pure methods.