New method separates objects from images using deep neural networks trained to inpaint.
problem Fully self-supervised instance separation of occluded objects in images.
method Maximizes independence of two image regions given a fully self-supervised inpainting network.
result Method achieves similar segmentation performance to fully supervised methods on microscopy image datasets.
A new neural network architecture reduces parameters by 94% while maintaining performance.
problem Reduction of trainable parameters in neural networks.
method Spatially-coupled sparse construction to allocate trainable parameters efficiently.
result Performance comparable to traditional neural networks with 94% fewer parameters.
New limits found for training deep learning models efficiently.
problem Optimizing the training speed of deep learning models without sacrificing accuracy.
method Applied stochastic thermodynamics to set speed limits for neural network training.
result Training neural networks is optimal within certain scaling assumptions.
Study shows different training methods yield varying prediction risks for two-layers neural networks.
problem Understanding prediction risks in two-layers neural networks under different training regimes.
method Examined three training regimes: random features, neural tangent, and fully trained neural network.
result There is a significant gap in prediction risk between the random features and neural tangent regimes when the number of neurons is smaller than the ambient dimension.
Convolutional networks outperform fully-connected ones in certain tasks.
problem Understanding the computational advantage of convolutional networks over fully-connected networks.
method Demonstrated a computational advantage through a specific problem class.
result Convolutional networks can solve certain problems that fully-connected networks cannot, even with gradient descent.
Bayesian neural networks benefit from fully marginalizing over all modes to improve generalization.
problem Bayesian neural networks suffer from multimodal posterior distributions that can lead to suboptimal generalization.
method Use appropriate Bayesian sampling tools to fully marginalize over all posterior modes.
result Training with full marginalization improves the ability of the network to reason between multiple candidate solutions.
We propose Sparse Neural Network architectures that are based on random or structured bipartite graph topologies. Sparse architectures provide compression of the models learned and speed-ups of computations, they can also surpass their unstructured or fully connected counterparts. As we show, even more compact topologi…
We study alignment in linear neural networks and its relation to gradient descent.
problem Understanding alignment in linear neural networks and its impact on training.
method Defined alignment for fully connected networks, analyzed alignment under gradient descent, and compared gradient descent to projected gradient descent for layer-constrained networks.
result Gradient descent can converge linearly to a global minimum when alignment is invariant, and alignment is impossible with large datasets in layer-constrained networks.
Study of deep neural networks' NTK evolution during training.
problem Understanding the performance gap between deep neural networks and kernel regression.
method Derive an infinite hierarchy of ordinary differential equations (NTH) to capture gradient descent dynamics of deep neural networks.
result Truncated NTH approximates the dynamic of the NTK up to arbitrary precision under certain conditions.
Fully convolutional neural networks (FCN) have been shown to achieve state-of-the-art performance on the task of classifying time series sequences. We propose the augmentation of fully convolutional networks with long short term memory recurrent neural network (LSTM RNN) sub-modules for time series classification. Our …
Bayesian Perceptron offers fully Bayesian neural networks without complex computations.
problem Lack of uncertainty quantification in neural networks.
method Bayesian inference framework for perceptron training and predictions in closed-form.
result Analytical expressions for perceptron's output and weight learning provided.
This paper shows overparameterized deep neural networks are convex and learn useful features.
problem Analyzing fully trained overparameterized deep neural networks.
method Generalized neural feature repopulation technique.
result Overparameterized deep neural networks are inherently convex and learn useful features.
A new RNN architecture reduces model size and improves performance.
problem Overparameterization and resource limitations in RNNs.
method Jointly encodes weight matrices using tensor-train factorization.
result Reduces model size by several orders of magnitude without sacrificing performance.
Orthogonal initialization does not speed up training in ultra-wide neural networks.
problem Exploring the effect of orthogonal initialization on training speed in deep neural networks.
method Study of neural tangent kernel dynamics in FCNs and CNNs with orthogonal initialization.
result The NTK of orthogonally-initialized networks remains constant during training, suggesting no speedup in the NTK regime.
DAERNN models censored data using neural networks with data augmentation.
problem Handling censored data in expectile regression.
method Data augmentation based Expectile Regression Neural Networks (ERNNs).
result DAERNN outperforms existing censored ERNNs methods and achieves comparable predictive performance to fully observed data.
USN improves neural networks with uniform sparse connectivity.
problem Overfitting and limited scalability in classical neural networks.
method Uniform sparse network (USN) with even and sparse connectivity.
result USN outperforms state-of-the-art sparse network models in accuracy, speed, and robustness.
It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of eva…
Network degeneracy affects training performance, especially in deep networks.
problem Degeneracy in deep neural networks leads to poor training performance.
method Predicted degeneracy level correlates with training dynamics using finite and infinite width networks.
result Degeneracy in neural networks correlates with training performance and can be predicted.
Efficient echo state network with explicit memory performs well on benchmark tasks.
problem Training differentiable neural computers is difficult and time-consuming.
method Echo state network with an explicit memory.
result Echo state network can recognize all regular languages, including those contractive networks cannot.
Gradient descent with growing learning rate enables learning non-linear features in neural networks.
problem Learning non-linear features in two-layer neural networks.
method Using gradient descent with a learning rate that grows with the sample size.
result Multiple rank-one components emerge, each corresponding to a specific polynomial feature.
Convex dual network improves neural network reconstruction for medical imaging.
problem Non-convex nature of neural networks hinders their use in sensitive applications.
method Introduces a convex duality framework for a two-layer fully-convolutional ReLU denoising network.
result Training neural networks with weight decay regularization induces path sparsity and piecewise linear filtering.
The paper examines how neural networks generalize in time series forecasting.
problem Time series data's non-i.i.d. nature affects neural network generalization.
method Used Hessians to quantify network generalization; controlled complexity with hyperparameters.
result Empirically validated generalization metrics for time series forecasting.
New algorithms reduce communication in GNN training.
problem Higher communication costs in GNNs due to sparse connectivity.
method Parallel algorithms for sparse-dense matrix multiplication.
result Asymptotic reduction in communication compared to previous methods.
This paper shows linear over-parametrization suffices for shallow neural networks to fit training data.
problem Training shallow neural networks with optimal over-parametrization.
method Used a simple variant of stochastic gradient descent.
result Linear over-parametrization is sufficient for shallow neural networks to fit training data.
New method approximates neural network training for robustness.
problem Training robust neural networks with adversarial input perturbations.
method Stochastic convex optimization approach to adversarial training.
result Method achieves better adversarial robustness and performance.
In our previous work we have shown that resistive cross point devices, so called Resistive Processing Unit (RPU) devices, can provide significant power and speed benefits when training deep fully connected networks as well as convolutional neural networks. In this work, we further extend the RPU concept for training re…
There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs). This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but b…
Recurrent Networks are one of the most powerful and promising artificial neural network algorithms to processing the sequential data such as natural languages, sound, time series data. Unlike traditional feed-forward network, Recurrent Network has a inherent feed back loop that allows to store the temporal context info…
Convolutional nets require fewer samples than fully-connected nets for image classification.
problem Understanding why convolutional nets are more sample-efficient than fully-connected nets.
method Construction of a natural distribution and target function to demonstrate a sample complexity gap.
result Convolutional nets require O(1) samples for a single target function, while fully-connected nets require Ω(d2) samples. This paper improves deep neural network approximation for fully connected networks, achieving optimal convergence rates.
problem Improving approximation of fully connected deep neural networks for optimal convergence rates.
method Deriving approximation bounds specifically for a narrower fully connected deep neural network.
result Achieves an optimal rate (up to a logarithmic factor) for fully connected deep neural networks.
NeuZip compresses neural network weights to save memory during training and inference.
problem Memory constraints in neural network training and inference.
method Entropy-based dynamic weight compression.
result Significant reduction in memory usage without performance loss.
A new method to simplify deep neural networks by removing unnecessary parts.
problem Overly complex deep neural networks require significant resource investment for size reduction.
method A fully differentiable sparsification method that optimizes a regularized objective function with stochastic gradient descent.
result The method can learn both the sparsified structure and weights of a network in an end-to-end manner.
This paper proposes a method to automatically compress neural networks using Bayesian tensor decomposition.
problem Challenges in directly applying tensor compression in neural network training.
method Bayesian tensorized neural network with automatic rank selection.
result Produces significantly more compact neural networks (7.4x to 137x) directly from training.
SIAN bridges simple models to neural networks by identifying necessary feature combinations.
problem The gap between simple models and powerful neural networks in performance.
method Feature interaction detection and sparse selection algorithm.
result Competitive performance across multiple tabular datasets with optimal tradeoff.
The fully connected layers of a deep convolutional neural network typically contain over 90% of the network parameters, and consume the majority of the memory required to store the network parameters. Reducing the number of parameters while preserving essentially the same predictive performance is critically important …
The ability of deep neural networks to generalize well in the overparameterized regime has become a subject of significant research interest. We show that overparameterized autoencoders exhibit memorization, a form of inductive bias that constrains the functions learned through the optimization process to concentrate a…
We present an instance segmentation algorithm trained and applied to a CCTV recording of beef cattle during a winter finishing period. A fully convolutional network was transformed into an instance segmentation network that learns to label each instance of an animal separately. We introduce a conceptually simple framew…
New framework explains why over-parameterized neural networks work well.
problem Why over-parameterized neural networks perform well in practice.
method Neural feature repopulation framework using gradient descent.
result Over-parameterized two-level neural networks learn near optimal feature distributions.
Training a neural network using backpropagation algorithm requires passing error gradients sequentially through the network. The backward locking prevents us from updating network layers in parallel and fully leveraging the computing resources. Recently, there are several works trying to decouple and parallelize the ba…
We embed KKT points in neural networks of different sizes.
problem Classifying data using homogeneous neural networks.
method Introducing KKT point embedding principle and proving it for different network types.
result KKT points of a smaller network can be mapped to those of a larger network via linear transformations.
Alternative neural network training using monotone variational inequality.
problem Training neural networks efficiently and with guarantees.
method Using monotone variational inequality to solve non-convex problems efficiently.
result Our approach leads to fast convergence and competitive performance compared to traditional methods.
Reduces CNN parameters by 2x with minimal loss in accuracy.
problem Parameter efficiency in distributed machine learning and edge devices.
method Introduces a Convolutional Slice Generator (CSG) to reduce CNN parameters.
result Reduces CNN parameters by approximately 2x with minimal accuracy loss.
Paper develops exact convex optimization for neural networks with polynomial activations.
problem Training two-layer neural networks with nonlinear polynomial activations.
method Exact convex optimization using semidefinite programming.
result Global optimization of neural networks is polynomial-time computable.
This informal technical report details the geometric illustration of decision boundaries for ReLU units in a three layer fully connected neural network. The network is designed and trained to predict pixel intensity from an (x, y) input location. The Geometric Illustration of Neural Networks (GINN) tool was built to vi…
The paper provides approximation guarantees for neural networks trained with gradient flow.
problem Approximating neural networks trained with gradient flow in continuous L2(Sd−1)-norm. method NTK argument for non-convex second but last layer, under-parametrized regime.
result Gradient flow convergence guarantees for neural networks under Sobolev smoothness assumptions.
MimosaNet prevents model stealing by making neural networks sensitive to weight changes.
problem Neural networks are vulnerable to model stealing due to robustness to minor parameter changes.
method Develops a method to create a sensitive version of a trained neural network.
result The sensitive network produces the same responses but is highly sensitive to weight changes, preventing model stealing.
We propose a new model for unsupervised document embedding. Leading existing approaches either require complex inference or use recurrent neural networks (RNN) that are difficult to parallelize. We take a different route and develop a convolutional neural network (CNN) embedding model. Our CNN architecture is fully par…
Efficient methods for training deep neural networks using subsampled Gauss-Newton and natural gradient.
problem Training deep neural networks with large datasets and variables.
method Subsampled Gauss-Newton and natural gradient methods with subsampled gradient estimates.
result Methods converge to a stationary point and are efficient to implement.