arXiv research

Boosts neural network performance by improving weight separability.

Gradient descent converges to perfect classification in neural nets for non-separable data.

Gradient flow on softmax attention minimizes nuclear norm of weight matrices.

Shallow nonlinear networks can separate classes linearly with polynomially scaling width.

Develops a Bayesian non-parametric approach for signal separation with varying components.

Importance-weighted risk minimization is a key ingredient in many machine learning algorithms for causal inference, domain adaptation, class imbalance, and off-policy reinforcement learning. While the effect of importance weighting is well-characterized for low-capacity misspecified models, little is known about how it…

A new method trains deep networks by separating weight locations from values.

Spectral analysis shows neural networks separate from linear methods in approximating functions.

Proposes a new binary classification model inspired by fluid phase separation.

Understanding separation effects on parameter estimation in finite Gaussian mixtures

Algorithm distinguishes Gaussian mixtures from pure Gaussians in quasi-polynomial time.

New findings on depth vs. width in neural networks, showing depth can improve learnability.

PRESTO improves rare event prediction by shrinking towards proportional odds model.

We show that deep networks can be trained using Hebbian updates yielding similar performance to ordinary back-propagation on challenging image datasets. To overcome the unrealistic symmetry in connections between layers, implicit in back-propagation, the feedback weights are separate from the feedforward weights. The f…

Constructs classifiers for neural networks with specific data configurations.

The paper proves barriers to approximating functions with small weights and depth in neural networks.

Algorithm clusters mixtures with bounded covariances under specific separation conditions.

Separating an audio scene into isolated sources is a fundamental problem in computer audition, analogous to image segmentation in visual scene analysis. Source separation systems based on deep learning are currently the most successful approaches for solving the underdetermined separation problem, where there are more …

Blind Source Separation is a widely used technique to analyze multichannel data. In many real-world applications, its results can be significantly hampered by the presence of unknown outliers. In this paper, a novel algorithm coined rGMCA (robust Generalized Morphological Component Analysis) is introduced to retrieve s…

We propose an algorithm to separate simultaneously speaking persons from each other, the "cocktail party problem", using a single microphone. Our approach involves a deep recurrent neural networks regression to a vector space that is descriptive of independent speakers. Such a vector space can embed empirically determi…

Paper proposes graph-based separable transforms for video coding.

EM algorithm achieves optimal sample complexity for well-separated Gaussian mixtures.

This research shows loss weighting remains effective in last layer retraining despite model overparameterization.

A new algorithm finds a separating hyperplane with fewer updates.

A new method corrects weight values to improve treatment effect estimation.

HydaLearn dynamically adjusts task weights for better MTL performance.

New algorithm recovers mixture means even with many outliers.

In importance sampling (IS)-based reinforcement learning algorithms such as Proximal Policy Optimization (PPO), IS weights are typically clipped to avoid large variance in learning. However, policy update from clipped statistics induces large bias in tasks with high action dimensions, and bias from clipping makes it di…

Study on optimal ReLU networks with weight decay for interpolation.

Let $f:\mathbb{S}^{d-1}\times \mathbb{S}^{d-1}\to\mathbb{S}$ be a function of the form $f(\mathbf{x},\mathbf{x}') = g(\langle\mathbf{x},\mathbf{x}'\rangle)$ for $g:[-1,1]\to \mathbb{R}$. We give a simple proof that shows that poly-size depth two neural networks with (exponentially) bounded weights cannot approximate $f…

CNNs require fewer samples than LCNs and FCNs for image-based tasks due to locality and weight sharing.

New summary measures reveal geometric structure in weighted measures on manifolds.

Deep convolutional networks provide state of the art classifications and regressions results over many high-dimensional problems. We review their architecture, which scatters data with a cascade of linear filter weights and non-linearities. A mathematical framework is introduced to analyze their properties. Computation…

Humans perceive the seemingly chaotic world in a structured and compositional way with the prerequisite of being able to segregate conceptual entities from the complex visual scenes. The mechanism of grouping basic visual elements of scenes into conceptual entities is termed as perceptual grouping. In this work, we pro…

Combining information from different sources is a common way to improve classification accuracy in Brain-Computer Interfacing (BCI). For instance, in small sample settings it is useful to integrate data from other subjects or sessions in order to improve the estimation quality of the spatial filters or the classifier. …

We investigate the problem of nodes clustering under privacy constraints when representing a dataset as a graph. Our contribution is threefold. First we formally define the concept of differential privacy for structured databases such as graphs, and give an alternative definition based on a new neighborhood notion betw…

Optimal ReLU networks can memorize any separable set of points with a small number of parameters.

For each cardinal $κ$, each natural number $n$ and each simplicial complex $K$ we construct a space $ν^n_κ(K)$ and a map $π\colon ν^n_κ(K) \to K$ such that the following conditions are satisfied. 1. $ν^n_κ(K)$ is a complete metric $n$-dimensional space of weight $κ$. 2. $ν^n_κ(K)$ is an absolute neighborhood extensor i…

The paper investigates how neural network weights evolve to monitor training progress.

We improve deep threshold networks' memorization capacity exponentially.

SPIRE enables efficient federated learning for diffusion models by separating client-specific embeddings from a shared backbone.

ETM identifies field-specific keywords in text classification.

Enhances linear regression with Kalman filter for loss minimization.

Biased sampling and missing data complicates statistical problems ranging from causal inference to reinforcement learning. We often correct for biased sampling of summary statistics with matching methods and importance weighting. In this paper, we study nearest neighbor matching (NNM), which makes estimates of populati…

In this paper, we propose a novel recurrent neural network architecture for speech separation. This architecture is constructed by unfolding the iterations of a sequential iterative soft-thresholding algorithm (ISTA) that solves the optimization problem for sparse nonnegative matrix factorization (NMF) of spectrograms.…

New algorithm learns multiclass concepts with finite Littlestone dimension.

A new text classification method using separable convolution reduces memory consumption.

We analyse multimodal time-series data corresponding to weight, sleep and steps measurements. We focus on predicting whether a user will successfully achieve his/her weight objective. For this, we design several deep long short-term memory (LSTM) architectures, including a novel cross-modal LSTM (X-LSTM), and demonstra…