arXiv research

Gradient span algorithms show consistent progress in high dimensions.

We introduce the "NoBackTrack" algorithm to train the parameters of dynamical systems such as recurrent neural networks. This algorithm works in an online, memoryless setting, thus requiring no backpropagation through time, and is scalable, avoiding the large computational and memory cost of maintaining the full gradie…

Abstract reviews algorithms for multi-index models, focusing on polynomial-time methods and their limitations.

Differentiable clustering method using perturbed spanning forests.

A new algorithm reduces bias and variance in distributionally robust optimization.

Unified framework for gradient estimation in combinatorial spaces.

We introduce a globally-convergent algorithm for optimizing the tree-reweighted (TRW) variational objective over the marginal polytope. The algorithm is based on the conditional gradient method (Frank-Wolfe) and moves pseudomarginals within the marginal polytope through repeated maximum a posteriori (MAP) calls. This m…

It has been observed in a variety of contexts that gradient descent methods have great success in solving low-rank matrix factorization problems, despite the relevant problem formulation being non-convex. We tackle a particular instance of this scenario, where we seek the $d$-dimensional subspace spanned by a streaming…

New algorithms find optimal policies without knowing MDP span.

A new algorithm estimates sparse gradients on graphs with improved risk bounds.

This paper explores adaptive methods in over-parameterized linear regression.

New method stabilizes saddle-point optimization with unbounded gradients.

Paper proposes recycling model updates in federated learning by exploiting low-rank gradient subspaces.

We use matrix iteration theory to characterize acceleration in smooth games. We define the spectral shape of a family of games as the set containing all eigenvalues of the Jacobians of standard gradient dynamics in the family. Shapes restricted to the real line represent well-understood classes of problems, like minimi…

Matching pursuit algorithms are an important class of algorithms in signal processing and machine learning. We present a blended matching pursuit algorithm, combining coordinate descent-like steps with stronger gradient descent steps, for minimizing a smooth convex function over a linear space spanned by a set of atoms…

A new classification method based on Minimum Spanning Trees

Weighted SVM (or fuzzy SVM) is the most widely used SVM variant owning its effectiveness to the use of instance weights. Proper selection of the instance weights can lead to increased generalization performance. In this work, we extend the span error bound theory to weighted SVM and we introduce effective hyperparamete…

New method for scalable barycenter computation using Wasserstein gradient flows.

Studying a softmax-attention model, we show that the learned query converges to the latent signal subspace spanned by the informative direction.

Paper proposes an algorithm to reconstruct optimal model structure from graph adjacency matrix.

We introduce SCAL, an algorithm designed to perform efficient exploration-exploitation in any unknown weakly-communicating Markov decision process (MDP) for which an upper bound $c$ on the span of the optimal bias function is known. For an MDP with $S$ states, $A$ actions and $Γ\leq S$ possible next states, we prove a …

A classification of spanning surfaces for alternating links is provided up to genus, orientability, and a new invariant that we call aggregate slope. That is, given an alternating link, we determine all possible combinations of genus, orientability, and aggregate slope that a surface spanning that link can have. To thi…

This paper learns state, dynamics, and filtering algorithms together for data assimilation.

New algorithms reduce reinforcement learning regret in factored MDPs.

This paper considers the problem of solving systems of quadratic equations, namely, recovering an object of interest $\mathbf{x}^{\natural}\in\mathbb{R}^{n}$ from $m$ quadratic equations/samples $y_{i}=(\mathbf{a}_{i}^{\top}\mathbf{x}^{\natural})^{2}$, $1\leq i\leq m$. This problem, also dubbed as phase retrieval, span…

We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of tr…

We investigate the problem of nodes clustering under privacy constraints when representing a dataset as a graph. Our contribution is threefold. First we formally define the concept of differential privacy for structured databases such as graphs, and give an alternative definition based on a new neighborhood notion betw…

We iterate Manolescu's unoriented skein exact triangle in knot Floer homology with coefficients in the field of rational functions over $\mathbb{Z}/2\mathbb{Z}$. The result is a spectral sequence which converges to a stabilized version of delta-graded knot Floer homology. The $(E_2,d_2)$ page of this spectral sequence …

Paper studies gradient fields from discrete Morse functions for watershed-cut computation.

FPCA optimizes fairness in target vectors' span.

Novel graph-spanning algorithm detects changes in high-dimensional data.

The family of temporal difference (TD) methods span a spectrum from computationally frugal linear methods like TD(λ) to data efficient least squares methods. Least square methods make the best use of available data directly computing the TD solution and thus do not require tuning a typically highly sensitive learning r…

A new hierarchical clustering method selects representative points from sub-minimum-spanning-trees.

Lumbermark clusters data robustly, slicing limbs of mutual reachability trees.

A new method for distributed optimization reduces communication rounds without minibatches.

We investigate the problem of sequentially predicting the binary labels on the nodes of an arbitrary weighted graph. We show that, under a suitable parametrization of the problem, the optimal number of prediction mistakes can be characterized (up to logarithmic factors) by the cutsize of a random spanning tree of the g…

Neural networks learn complex functions efficiently near information-theoretic limits.

cuSLINK clusters data faster on GPUs, saving space and time.

A new method for optimizing models with categorical variables using diffusion.

MSTs provide a fast and meaningful clustering method in low-dimensional data.

We introduce a new class of lower bounds on the log partition function of a Markov random field which makes use of a reversed Jensen's inequality. In particular, our method approximates the intractable distribution using a linear combination of spanning trees with negative weights. This technique is a lower-bound count…

We consider the problem of recovering low-rank matrices from random rank-one measurements, which spans numerous applications including covariance sketching, phase retrieval, quantum state tomography, and learning shallow polynomial neural networks, among others. Our approach is to directly estimate the low-rank factor …

This note gives the first example of a hyperbolic knot in the 3-sphere that lacks a nonorientable essential spanning surface; this disproves the Strong Neuwirth Conjecture formulated by Ozawa and Rubinstein. Moreover, this knot has no even strict boundary slopes, disproving the Even Boundary Slope Conjecture of the sam…

Gradient equilibrium improves online learning performance without requiring sublinear regret.

We consider the detection of activations over graphs under Gaussian noise, where signals are piece-wise constant over the graph. Despite the wide applicability of such a detection algorithm, there has been little success in the development of computationally feasible methods with proveable theoretical guarantees for ge…

Subspace learning and matrix factorization problems have great many applications in science and engineering, and efficient algorithms are critical as dataset sizes continue to grow. Many relevant problem formulations are non-convex, and in a variety of contexts it has been observed that solving the non-convex problem d…

FedLoRU improves FL efficiency by using low-rank updates.

We study the problem of recovering the subspace spanned by the first $k$ principal components of $d$-dimensional data under the streaming setting, with a memory bound of $O(kd)$. Two families of algorithms are known for this problem. The first family is based on the framework of stochastic gradient descent. Nevertheles…