arXiv research

Simply connected spaces of tight frames identified.

Kernel adaptive filters, a class of adaptive nonlinear time-series models, are known by their ability to learn expressive autoregressive patterns from sequential data. However, for trivial monotonic signals, they struggle to perform accurate predictions and at the same time keep computational complexity within desired …

Gradient descent constructs tight fusion frames.

A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary. This ensures eigenvalues with unit norm and thus stable dynamics and training. However …

When geometric structures on surfaces are determined by the lengths of curves, it is natural to ask: which curves' lengths do we really need to know? It is a result of Duchin--Leininger--Rafi that any flat metric induced by a unit-norm quadratic differential is determined by its marked simple length spectrum. We genera…

We present a method for training multi-label, massively multi-class image classification models, that is faster and more accurate than supervision via a sigmoid cross-entropy loss (logistic regression). Our method consists in embedding high-dimensional sparse labels onto a lower-dimensional dense sphere of unit-normed …

Quantizes neural networks using frame theory for improved accuracy.

This note presents an analytic construction of the optimal unit-norm direction hat(x) = x/|x| that maximizes or minimizes the objective linear expression, B . hat(x), subject to a system of linear constraints of the form [A] . x = 0, where x is an unknown n-dimensional real vector to be determined up to an overall norm…

The paper studies matrix normalization and graph balancing using a new functional and gradient descent.

The paper constructs a lamination related to minimal hypersurfaces calibrated by a cohomology class.

We propose a Generalized Dantzig Selector (GDS) for linear models, in which any norm encoding the parameter structure can be leveraged for estimation. We investigate both computational and statistical aspects of the GDS. Based on conjugate proximal operator, a flexible inexact ADMM framework is designed for solving GDS…

Representing shapes as level sets of neural networks has been recently proved to be useful for different shape analysis and reconstruction tasks. So far, such representations were computed using either: (i) pre-computed implicit shape representations; or (ii) loss functions explicitly defined over the neural level sets…

We consider a closed orientable Riemannian 3-manifold $(M,g)$ and a vector field $X$ with unit norm whose integral curves are geodesics of $g$. Any such vector field determines naturally a 2-plane bundle contained in the kernel of the contact form of the geodesic flow of $g$. We study when this 2-plane bundle remains i…

We give a polynomial-time algorithm for learning neural networks with one layer of sigmoids feeding into any Lipschitz, monotone activation function (e.g., sigmoid or ReLU). We make no assumptions on the structure of the network, and the algorithm succeeds with respect to {\em any} distribution on the unit ball in $n$ …

Regularization improves stability and consistency of sparse autoencoders.

New method relaxes PCA orthogonality constraints using explained variance of correlated components.

FF algorithm uses goodness as a likelihood-ratio test for scalar normalization.

Analysis of non-asymptotic estimation error and structured statistical recovery based on norm regularized regression, such as Lasso, needs to consider four aspects: the norm, the loss function, the design matrix, and the noise model. This paper presents generalizations of such estimation error analysis on all four aspe…

FF algorithm uses goodness as a measure of input quality, derived from likelihood-ratio tests.

Study analyzes accuracy of tensor deflation in noisy conditions.

We address the problem of defining a group sparse formulation for Principal Components Analysis (PCA) - or its equivalent formulations as Low Rank approximation or Dictionary Learning problems - which achieves a compromise between maximizing the variance explained by the components and promoting sparsity of the loading…

We present Rotated Adaptive Tetra-iterated Quantizer (RATQ), a fixed-length quantizer for gradients in first order stochastic optimization. RATQ is easy to implement and involves only a Hadamard transform computation and adaptive uniform quantization with appropriately chosen dynamic ranges. For noisy gradients with al…

Study uses Wasserstein distance to identify causal orders and unmix sources.

AuON is a linear-time optimizer that improves upon Muon's performance without approximate orthogonal matrices.

This work develops machine learning for micromagnetic energy minimization.

While deep learning is successful in a number of applications, it is not yet well understood theoretically. A satisfactory theoretical characterization of deep learning however, is beginning to emerge. It covers the following questions: 1) representation power of deep networks 2) optimization of the empirical risk 3) g…

Improved online PCA algorithm learns from evolving norm of parameter vector.

We study the computational cost of recovering a unit-norm sparse principal component $x \in \mathbb{R}^n$ planted in a random matrix, in either the Wigner or Wishart spiked model (observing either $W + λxx^\top$ with $W$ drawn from the Gaussian orthogonal ensemble, or $N$ independent samples from $\mathcal{N}(0, I_n + …

A new method to improve deep neural networks using weight rescaling.

New algorithm calibrates forecasts in high dimensions with minimal regret.