arXiv research

A new algorithm trains deep neural networks by adding neurons greedily.

Can multilayer neural networks -- typically constructed as highly complex structures with many nonlinearly activated neurons across layers -- behave in a non-trivial way that yet simplifies away a major part of their complexities? In this work, we uncover a phenomenon in which the behavior of these complex networks -- …

The study examines the distribution of projections of Gaussian data points and its implications for learning models.

Isometry pursuit identifies orthonormal submatrices from wide matrices.

We study a form of cyclic pursuit on Riemannian manifolds with positive injectivity radius. We conjecture that on a compact manifold, the piecewise geodesic loop formed by connecting consecutive pursuit agents either collapses in finite time or converges to a closed geodesic. The main result is that this conjecture is …

Study of pursuit-evasion game on sphere and its relation to planar Apollonius circle.

This paper closes the gap on matching pursuit's convergence rate.

Pursuit-evasion is a multi-agent sequential decision problem wherein a group of agents known as pursuers coordinate their traversal of a spatial domain to locate an agent trying to evade them. Pursuit evasion problems arise in a number of import application domains including defense and route planning. Learning to opti…

Two popular examples of first-order optimization methods over linear spaces are coordinate descent and matching pursuit algorithms, with their randomized variants. While the former targets the optimization by moving along coordinates, the latter considers a generalized notion of directions. Exploiting the connection be…

Generative ML learns optimal pursuit trajectories in pursuit-evasion games.

Matching pursuit algorithms are an important class of algorithms in signal processing and machine learning. We present a blended matching pursuit algorithm, combining coordinate descent-like steps with stronger gradient descent steps, for minimizing a smooth convex function over a linear space spanned by a set of atoms…

The paper finds non-Gaussian directions in high-dimensional data using Wasserstein distance.

In this correspondence, we obtain exact recovery conditions for regularized modified basis pursuit (reg-mod-BP) and discuss when the obtained conditions are weaker than those for modified-CS or for basis pursuit (BP). The discussion is also supported by simulation comparisons. Reg-mod-BP provides a solution to the spar…

Insurance firms use RL to optimize customer offers for desired target portfolios.

Enhances projection pursuit tree classifier with visual diagnostics for better multi-class classification.

Product models of low dimensional experts are a powerful way to avoid the curse of dimensionality. We present the ``under-complete product of experts' (UPoE), where each expert models a one dimensional projection of the data. The UPoE is fully tractable and may be interpreted as a parametric probabilistic model for pro…

Sparse Subspace Clustering (SSC) is a state-of-the-art method for clustering high-dimensional data points lying in a union of low-dimensional subspaces. However, while $\ell_1$ optimization-based SSC algorithms suffer from high computational complexity, other variants of SSC, such as Orthogonal Matching Pursuit-based S…

Recently, the principal component pursuit has received increasing attention in signal processing research ranging from source separation to video surveillance. So far, all existing formulations are real-valued and lack the concept of phase, which is inherent in inputs such as complex spectrograms or color images. Thus,…

Orthogonal Matching Pursuit (OMP) has long been considered a powerful heuristic for attacking compressive sensing problems; however, its theoretical development is, unfortunately, somewhat lacking. This paper presents an improved Restricted Isometry Property (RIP) based performance guarantee for T-sparse signal reconst…

Sharp results link DLN gradient flow to basis pursuit optimization and GHA phase transitions.

Two of the most fundamental prototypes of greedy optimization are the matching pursuit and Frank-Wolfe algorithms. In this paper, we take a unified view on both classes of methods, leading to the first explicit convergence rates of matching pursuit methods in an optimization sense, for general sets of atoms. We derive …

Paper proposes a novel approach to density ratio estimation using projection pursuit.

The problem of pursuing a moving target is always one of the main topics in navigation. In the literatures, there are two well-known algorithms called Pure Pursuit and Pure Rendezvous navigation in the 3-dimensional space $\mathbb{R}^3$. In this paper, these two methods are combined to introduce a novel family of pursu…

A geometric approach to differential game theory is illustrated. The parallel pursuit is considered as a two-player zero-sum differential game. The optimal strategies of each player is designed based on Riemann-Finsler geometry. Our approach incorporates a closed loop optimal control and the presentation is familiar wi…

Projection pursuit model improves Gaussian process regression for high-dimensional data.

In this paper, we explore the limitations of PCA as a dimension reduction technique and study its extension, projection pursuit (PP), which is a broad class of linear dimension reduction methods. We first discuss the relevant concepts and theorems and then apply PCA and PP (with negative standardized Shannon's entropy …

OOMP selects features online for sparse linear regression.

Paper proposes PPMM for fast estimation of large-scale OTM.

This paper presents a new ensemble learning method for classification problems called projection pursuit random forest (PPF). PPF uses the PPtree algorithm introduced in Lee et al. (2013). In PPF, trees are constructed by splitting on linear combinations of randomly chosen variables. Projection pursuit is used to choos…

Lower bounds show OLS outperforms basis pursuit in overparameterized linear regression.

We introduce a new convex formulation for stable principal component pursuit (SPCP) to decompose noisy signals into low-rank and sparse representations. For numerical solutions of our SPCP formulation, we first develop a convex variational framework and then accelerate it with quasi-Newton methods. We show, via synthet…

Unmanned Aerial Vehicles (UAVs), autonomously-guided aircraft, are widely used for tasks involving surveillance and reconnaissance. A version of the pursuit-evasion problems centered around UAVs and its variants has been extensively studied in recent years due to numerous breakthroughs in AI. We present an approach to …

The recently proposed Multi-Layer Convolutional Sparse Coding (ML-CSC) model, consisting of a cascade of convolutional sparse layers, provides a new interpretation of Convolutional Neural Networks (CNNs). Under this framework, the computation of the forward pass in a CNN is equivalent to a pursuit algorithm aiming to e…

Parsimonious representations are ubiquitous in modeling and processing information. Motivated by the recent Multi-Layer Convolutional Sparse Coding (ML-CSC) model, we herein generalize the traditional Basis Pursuit problem to a multi-layer setting, introducing similar sparse enforcing penalties at different representat…

Neuron Shapley identifies key neurons in deep networks, improving model accuracy and fairness.

Paper develops efficient AltMin algorithm for SRPCP robust matrix recovery.

Sparse Subspace Clustering (SSC) is one of the most popular methods for clustering data points into their underlying subspaces. However, SSC may suffer from heavy computational burden. Orthogonal Matching Pursuit applied on SSC accelerates the computation but the trade-off is the loss of clustering accuracy. In this pa…

Describes explaining neurons in deep representations using compositional logical concepts.

SeReNe prunes neurons with low sensitivity to reduce network size.

Under-parameterized networks can either copy or average teacher weights, leading to universal optimal solutions.

Matching Pursuit LASSIn Part I \cite{TanPMLPart1}, a Matching Pursuit LASSO ({MPL}) algorithm has been presented for solving large-scale sparse recovery (SR) problems. In this paper, we present a subspace search to further improve the performance of MPL, and then continue to address another major challenge of SR -- bat…

Modeling hidden neurons in SNNs using mesoscopic approximations.

Topological methods improve neuron analysis and tracer injection summary.

Efficiently representing real world data in a succinct and parsimonious manner is of central importance in many fields. We present a generalized greedy pursuit framework, allowing us to efficiently solve structured matrix factorization problems, where the factors are allowed to be from arbitrary sets of structured vect…

Inspired by complexity and diversity of biological neurons, our group proposed quadratic neurons by replacing the inner product in current artificial neurons with a quadratic operation on input data, thereby enhancing the capability of an individual neuron. Along this direction, we are motivated to evaluate the power o…

Informed by recent work on tensor singular value decomposition and circulant algebra matrices, this paper presents a new theoretical bridge that unifies the hypercomplex and tensor-based approaches to singular value decomposition and robust principal component analysis. We begin our work by extending the principal comp…

We propose a new generic type of stochastic neurons, called $q$-neurons, that considers activation functions based on Jackson's $q$-derivatives with stochastic parameters $q$. Our generalization of neural network architectures with $q$-neurons is shown to be both scalable and very easy to implement. We demonstrate expe…

Paper analyzes and improves GPSP algorithm for block sparse signal recovery.