arXiv research

Study evaluates relevance metrics for similarity-based model explanations.

AdEMAMix optimizer improves model performance and convergence speed.

We propose in this contribution a method for l one regularization in prototype based relevance learning vector quantization (LVQ) for sparse relevance profiles. Sparse relevance profiles in hyperspectral data analysis fade down those spectral bands which are not necessary for classification. In particular, we consider …

Deep neural networks are usually trained with stochastic gradient descent (SGD), which minimizes objective function using very rough approximations of gradient, only averaging to the real gradient. Standard approaches like momentum or ADAM only consider a single direction, and do not try to model distance from extremum…

Data repetition improves SGD's learning of high-dimensional functions.

TrIM improves gradient-based dimension reduction and regression.

This paper studies the complexity of the stochastic gradient algorithm for PCA when the data are observed in a streaming setting. We also propose an online approach for selecting the learning rate. Simulation experiments confirm the practical relevance of the plain stochastic gradient approach and that drastic improvem…

Neural networks can learn useful representations that kernels can't.

Proposes a new adaptive gradient method based on gradient differences.

Recently, a technique called Layer-wise Relevance Propagation (LRP) was shown to deliver insightful explanations in the form of input space relevances for understanding feed-forward neural network classification decisions. In the present work, we extend the usage of LRP to recurrent neural networks. We propose a specif…

In this paper we show how techniques coming from stochastic analysis, such as stochastic completeness (in the form of the weak maximum principle at infinity), parabolicity and $L^p$-Liouville type results for the weighted Laplacian associated to the potential may be used to obtain triviality, rigidity results, and scal…

Paper develops methods for statistical inference in SGD with infinite variance.

Data sets for fairness relevant tasks can lack examples or be biased according to a specific label in a sensitive attribute. We demonstrate the usefulness of weight based meta-learning approaches in such situations. For models that can be trained through gradient descent, we demonstrate that there are some parameter co…

Traditional model-based reinforcement learning approaches learn a model of the environment dynamics without explicitly considering how it will be used by the agent. In the presence of misspecified model classes, this can lead to poor estimates, as some relevant available information is ignored. In this paper, we introd…

Paper studies gradient fields from discrete Morse functions for watershed-cut computation.

Estimates gradients of solutions on closed surfaces.

Paper analyzes dataset distillation for efficient encoding of task-relevant information.

There is a long history of using meta learning as representation learning, specifically for determining the relevance of inputs. In this paper, we examine an instance of meta-learning in which feature relevance is learned by adapting step size parameters of stochastic gradient descent---building on a variety of prior w…

We wish to compute the gradient of an expectation over a finite or countably infinite sample space having $K \leq \infty$ categories. When $K$ is indeed infinite, or finite but very large, the relevant summation is intractable. Accordingly, various stochastic gradient estimators have been proposed. In this paper, we de…

DeepUnHide uses deep learning to reveal hidden demographic features in recommender systems.

Undergraduate thesis explores topological barriers to compact Ricci solitons in 4D.

NES optimizes discrete structured VAEs effectively without gradient propagation.

Paper analyzes SGHMC for non-convex optimization with discontinuous gradients.

We introduce local expectation gradients which is a general purpose stochastic variational inference algorithm for constructing stochastic gradients through sampling from the variational distribution. This algorithm divides the problem of estimating the stochastic gradients over multiple variational parameters into sma…

We cast Amari's natural gradient in statistical learning as a specific case of Kalman filtering. Namely, applying an extended Kalman filter to estimate a fixed unknown parameter of a probabilistic model from a series of observations, is rigorously equivalent to estimating this parameter via an online stochastic natural…

This paper proposes a new framework to regularize the highly ill-posed and non-linear phase retrieval problem through deep generative priors using simple gradient descent algorithm. We experimentally show effectiveness of proposed algorithm for random Gaussian measurements (practically relevant in imaging through scatt…

Contrastive learning struggles with class collapse and feature suppression, revealing bias towards simpler solutions.

Machine learning models predict US economic recessions using Treasury term spreads.

Neural network learning is usually time-consuming since backpropagation needs to compute full gradients and backpropagate them across multiple layers. Despite its success of existing works in accelerating propagation through sparseness, the relevant theoretical characteristics remain under-researched and empirical stud…

Attribution methods aim to explain a neural network's prediction by highlighting the most relevant image areas. A popular approach is to backpropagate (BP) a custom relevance score using modified rules, rather than the gradient. We analyze an extensive set of modified BP methods: Deep Taylor Decomposition, Layer-wise R…

New neural networks learn single-index models efficiently.

Gradient-based methods improve understanding of deep learning survival models.

This monograph presents the main complexity theorems in convex optimization and their corresponding algorithms. Starting from the fundamental theory of black-box optimization, the material progresses towards recent advances in structural optimization and stochastic optimization. Our presentation of black-box optimizati…

A feature selection algorithm should ideally satisfy four conditions: reliably extract relevant features; be able to identify non-linear feature interactions; scale linearly with the number of features and dimensions; allow the incorporation of known sparsity structure. In this work we propose a novel feature selection…

Gradient Starvation causes neural networks to focus on a subset of features, leading to imbalance.

We propose a novel Shapley value approach to help address neural networks' interpretability and "vanishing gradient" problems. Our method is based on an accurate analytical approximation to the Shapley value of a neuron with ReLU activation. This analytical approximation admits a linear propagation of relevance across …

Paper analyzes convergence of dynamic policy gradient for MDPs, improving performance in finite-time problems.

We formulate a general framework for competitive gradient-based learning that encompasses a wide breadth of multi-agent learning algorithms, and analyze the limiting behavior of competitive gradient-based learning algorithms using dynamical systems theory. For both general-sum and potential games, we characterize a non…

A new path gradient estimator speeds up normalizing flows without sacrificing accuracy.

New method predicts neural network performance using free probability theory.

Analyzes self-attention in recurrent networks, proving it mitigates vanishing gradients.

It has been observed in a variety of contexts that gradient descent methods have great success in solving low-rank matrix factorization problems, despite the relevant problem formulation being non-convex. We tackle a particular instance of this scenario, where we seek the $d$-dimensional subspace spanned by a streaming…

Gradient descent promotes low-rank solutions in tensor completion.

Energy price forecasting is a relevant yet hard task in the field of multi-step time series forecasting. In this paper we compare a well-known and established method, ARMA with exogenous variables with a relatively new technique Gradient Boosting Regression. The method was tested on data from Global Energy Forecasting …

Gradient descent struggles to learn equivariant neural networks, even with symmetries.

Improved saliency maps for deep neural networks with reduced noise.

We introduce a natural extension of the concept of gradient Ricci soliton: the Ricci almost soliton. We provide existence and rigidity results, we deduce a-priori curvature estimates and isolation phenomena, and we investigate some topological properties. A number of differential identities involving the relevant geome…

The paper shows how data and algorithm interactions affect overparameterized linear regression generalization.