arXiv research

New method reduces overfitting in deep neural networks by measuring and regulating hidden unit diversity.

Wide hidden layer TCM nets capacity analyzed using RDT and fl RDT.

New analysis shows capacity of treelike neural networks with various activations.

Optimal hidden-target learning for online inventory optimization on general convex sets.

We study finite sample expressivity, i.e., memorization power of ReLU networks. Recent results require $N$ hidden nodes to memorize/interpolate arbitrary $N$ data points. In contrast, by exploiting depth, we show that 3-layer ReLU networks with $Ω(\sqrt{N})$ hidden nodes can perfectly memorize most datasets with $N$ po…

New bounds on neural network capacity for treelike sign perceptrons using RDT.

Improved neural network capacity analysis using simplified RDT.

Neural networks learn more efficiently with hidden factorial structures.

Deep networks don't improve on shallow ones for finding minima.

Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous …

Financial markets are not random, but hard to predict due to hidden causes and strategic use.

Many state-of-the-art results obtained with deep networks are achieved with the largest models that could be trained, and if more computation power was available, we might be able to exploit much larger datasets in order to improve generalization ability. Whereas in learning algorithms such as decision trees the ratio …

We explore a framework called boosted Markov networks to combine the learning capacity of boosting and the rich modeling semantics of Markov networks and applying the framework for video-based activity recognition. Importantly, we extend the framework to incorporate hidden variables. We show how the framework can be ap…

Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and pe…

Chemical networks outperform spiking neural networks in classification tasks.

Normalization layers control deep neural network capacity, improving stability and generalization.

GenHMM combines neural networks with HMM for sequential data modeling.

This work presents entropic constraints from DAGs with hidden variables.

Combining Bayesian nonparametrics and a forward model selection strategy, we construct parsimonious Bayesian deep networks (PBDNs) that infer capacity-regularized network architectures from the data and require neither cross-validation nor fine-tuning when training the model. One of the two essential components of a PB…

Analyzes how ReLU affects neural network capacity and solution space geometry.

Matryoshka hides secret models in a carrier model, achieving high capacity and robustness.

A general Boltzmann machine with continuous visible and discrete integer valued hidden states is introduced. Under mild assumptions about the connection matrices, the probability density function of the visible units can be solved for analytically, yielding a novel parametric density function involving a ratio of Riema…

Differential forms and symmetric tensors show contrasting singular behaviors in a specific geometric setting.

Reservoir Computing (RC) refers to a Recurrent Neural Networks (RNNs) framework, frequently used for sequence learning and time series prediction. The RC system consists of a random fixed-weight RNN (the input-hidden reservoir layer) and a classifier (the hidden-output readout layer). Here we focus on the sequence lear…

Deep dynamic generative models are developed to learn sequential dependencies in time-series data. The multi-layered model is designed by constructing a hierarchy of temporal sigmoid belief networks (TSBNs), defined as a sequential stack of sigmoid belief networks (SBNs). Each SBN has a contextual hidden state, inherit…

Understanding the representational power of Restricted Boltzmann Machines (RBMs) with multiple layers is an ill-understood problem and is an area of active research. Motivated from the approach of \emph{Inherent Structure formalism} (Stillinger & Weber, 1982), extensively used in analysing Spin Glasses, we propose a no…

New method selects relevant dimensions for better prediction in mixtures.

We develop a novel method for training of GANs for unsupervised and class conditional generation of images, called Linear Discriminant GAN (LD-GAN). The discriminator of an LD-GAN is trained to maximize the linear separability between distributions of hidden representations of generated and targeted samples, while the …

Hidden Markov Models classify cough events with high accuracy.

Deep ReLU networks show that 4 layers suffice for unique input recovery.

How to model distribution of sequential data, including but not limited to speech and human motions, is an important ongoing research problem. It has been demonstrated that model capacity can be significantly enhanced by introducing stochastic latent variables in the hidden states of recurrent neural networks. Simultan…

Wide neural networks can degrade performance, contrary to conventional wisdom.

DEVDAN adapts to changing data streams by dynamically adding and removing hidden units.

Secret neural networks hidden within trained models.

Improved classifier accuracy by using more of the class-specific structure in trained models.

Proposes M-CHMM for robust modeling of multivariate healthcare time series.

Neural networks outperform kernels by learning features better.

Mean field Gaussian inference limits mutual information to regularize neural networks.

A new deep learning framework captures multi-scale spatio-temporal dependencies.

We study the problem of large scale, multi-label visual recognition with a large number of possible classes. We propose a method for augmenting a trained neural network classifier with auxiliary capacity in a manner designed to significantly improve upon an already well-performing model, while minimally impacting its c…

Funnel-Transformer reduces computation by compressing sequence data.

New neural network approach solves Poisson equations efficiently.

In this paper, we introduce a novel method to interpret recurrent neural networks (RNNs), particularly long short-term memory networks (LSTMs) at the cellular level. We propose a systematic pipeline for interpreting individual hidden state dynamics within the network using response characterization methods. The ranked …

LS4 models time-series with latent states, outperforming previous methods.

NADINE builds MLPs from streaming data, overcoming forgetting issues.

UWM-JEPA predicts future scenarios in belief space, improving accuracy in partially observed environments.

Paper presents a method for probabilistic load forecasting using adaptive online learning.

Network analysis reveals hidden user preferences for platform steering.