Dense neural networks can't approximate all functions.
problem Approximation capabilities of dense neural networks.
method Model compression approach combining weak regularity lemma and graph neural networks.
result Existence of Lipschitz continuous functions not approximable by dense neural networks.
The key idea of current deep learning methods for dense prediction is to apply a model on a regular patch centered on each pixel to make pixel-wise predictions. These methods are limited in the sense that the patches are determined by network architecture instead of learned from data. In this work, we propose the dense…
This paper explores loss landscapes of sparse neural networks, finding unique characteristics compared to dense networks.
problem Understanding the loss landscape of sparse neural networks, especially one-hidden-layer networks.
method Analyzes sparse networks with dense and sparse final layers, focusing on linear and non-linear models.
result Sparse networks can have no spurious valleys under certain conditions, but spurious valleys and minima can exist for wide sparse networks.
Neural networks are dense among Lipschitz functions with fixed Lipschitz constant.
problem Characterizing neural network approximations to Lipschitz functions.
method Analyzing L-Lipschitz neural networks and their density in L-Lipschitz functions. result One layer neural networks are dense in the set of all L-Lipschitz functions. RDL-Net improves speech enhancement with fewer parameters and better performance.
problem Improving speech enhancement with fewer parameters and better performance.
method Proposes RDL-Net, a CNN combining residual and dense aggregations without over-allocating parameters.
result RDL-Net achieves higher speech enhancement performance with fewer parameters and lower computational requirements.
Enhanced image denoising with MWRDCNN using residual dense blocks.
problem Image denoising with improved performance and robustness.
method Multi-wavelet residual dense convolutional neural network (MWRDCNN) with residual dense blocks (RDBs).
result Significantly improved performance in image denoising compared to existing techniques.
New framework for dense weighted networks with community-specific patterns.
problem Dense networks with varying edge weights across communities.
method Proposes a new model with functions mapping node characteristics to edge weights, requiring few parameters.
result Developed a bootstrap methodology for generating new networks.
Generative model captures hubs and dense communities in social networks.
problem Capturing both hubs and dense communities in social networks.
method Graphon mixture model with a new condition on sparse graphs.
result Estimation of hub normalized degree and graphon for sparse components.
Method generates dense fields from sparse measurements without needing spatial statistics or examples.
problem Generating dense physical fields from sparse measurements.
method Introduces a differentiable numerical simulator into neural network training.
result Superior results on fluid mechanics problems compared to statistical and neural network methods.
Polynomial-time test for detecting dense subgraphs in heterogeneous networks.
problem Detecting a planted community in heterogeneous networks.
method Proposes a polynomial-time test with a standard normal distribution null limiting distribution.
result The test is efficient and performs well in both simulations and real data.
We address feature interpretation and reproducibility issues in dense nets, proposing a modified loss function.
problem Feature interpretation and reproducibility issues in dense nets.
method Proposed a modified loss function to circumvent basis collapse.
result Substantially concise nets with 100x fewer parameters and lower MSE loss.
Sparse butterfly network replaces dense layers in neural networks, improving expressibility and performance.
problem Improving expressibility and performance of neural networks with dense layers.
method Replacing dense layers with a butterfly network architecture.
result The proposed architecture significantly reduces the number of weights from quadratic to nearly linear, with comparable or better performance.
SpaPool combines dense and sparse techniques for efficient graph pooling.
problem Efficiently processing large graphs in graph neural networks.
method Adaptive clustering of graph vertices into clusters.
result SpaPool outperforms existing methods on small-scale graphs.
Most artificial networks today rely on dense representations, whereas biological networks rely on sparse representations. In this paper we show how sparse representations can be more robust to noise and interference, as long as the underlying dimensionality is sufficiently high. A key intuition that we develop is that …
Bayesian networks are typically faithful, with implications for causal inference.
problem Determining the typicality of faithfulness in Bayesian networks.
method Analysis of Bayesian networks over a given DAG, parametrized by conditional exponential families, and nonparametric conditional densities.
result The faithful Bayesian networks are dense and open with respect to the total variation metric, extending existing results for specific classes of Bayesian networks.
Develops hyperparameter transfer methods for Dense Associative Memories.
problem Challenges in transferring hyperparameters for DenseAMs due to unique architecture and activation functions.
method Derives explicit prescriptions for hyperparameter transfer from small to large models.
result Excellent agreement between theoretical and empirical results.
Recent advances in neuroscience have revealed many principles about neural processing. In particular, many biological systems were found to reconfigure/recruit single neurons to generate multiple kinds of decisions. Such findings have the potential to advance our understanding of the design and optimization process of …
Paper develops Dense NN models for temporal-spatial data with improved performance.
problem Improving predictive performance and robustness in temporal-spatial modeling.
method Fully connected neural networks with ReLU activation, non-asymptotic bounds, manifold modeling, short-range dependence.
result Demonstrates superior performance in temporal-spatial modeling across various synthetic functions.
FuseInit optimizes neural network initialization by fusing layers from deeper networks.
problem Initialization sensitivity of neural networks.
method Fusing neighboring layers from deeper networks trained with random initialization.
result Shallower networks initialized with FuseInit can perform as well as deeper networks.
Tensor Neural Networks improve pricing accuracy for interest rate derivatives.
problem Inaccurate pricing of Bermudan Swaptions using traditional methods.
method Leveraging Tensor Neural Networks to solve backward Stochastic Differential Equations.
result Tensor Neural Networks provide more accurate and robust prices than Dense Neural Networks.
We study the problem of large-scale network embedding, which aims to learn latent representations for network mining applications. Previous research shows that 1) popular network embedding benchmarks, such as DeepWalk, are in essence implicitly factorizing a matrix with a closed form, and 2)the explicit factorization o…
Convolutional neural network (CNN) and its variants have led to many state-of-art results in various fields. However, a clear theoretical understanding about them is still lacking. Recently, multi-layer convolutional sparse coding (ML-CSC) has been proposed and proved to equal such simply stacked networks (plain networ…
New neural model processes 2D data with long-range dependencies efficiently.
problem Limited receptive field of convolutions for complex 2D tasks.
method Proposes Matrix Shuffle-Exchange network with O(logn) layers and O(n2logn) complexity. result Exceeds convolutional and graph neural network baselines in long-range dependency modeling.
Study on detecting and recovering hidden dense cycles in random graphs.
problem Detecting and recovering hidden dense cycles in random graphs.
method Information-theoretic analysis of thresholds for detection and recovery.
result Characterization of information-theoretic thresholds for detection and recovery.
Sparse Vision MoE matches dense networks in image recognition while using less compute.
problem Scaling vision models efficiently in computer vision.
method Vision MoE (V-MoE) - a sparse version of Vision Transformer.
result V-MoE matches state-of-the-art dense networks in image recognition with half the compute.
ARMA nets expand receptive fields for dense prediction tasks.
problem Global information in dense prediction problems is challenging for traditional convolutional layers.
method ARMA layers with adjustable autoregressive coefficients replace traditional convolutions.
result ARMA networks improve dense prediction tasks including video prediction and semantic segmentation.
The sizes of deep neural networks (DNNs) are rapidly outgrowing the capacity of hardware to store and train them. Research over the past few decades has explored the prospect of sparsifying DNNs before, during, and after training by pruning edges from the underlying topology. The resulting neural network is known as a …
New GPU kernels boost deep learning speed and memory efficiency.
problem Sparse deep learning matrices are not well-suited for existing sparse kernels.
method Identified favorable properties of sparse matrices from deep learning, developed high-performance GPU kernels for sparse matrix operations.
result 27% of single-precision peak performance on Nvidia V100 GPUs achieved with new kernels.
Dense Associative Memories outperform classical networks in robustness and signal processing.
problem Improving neural network performance in adversarial attacks and weak signal processing.
method Relaxing replica symmetry in statistical mechanics of spin glasses to analyze unsupervised and supervised learning.
result Explicit analytical investigation of phase diagrams and storage capacities for Dense Associative Memories.
The recent application of deep learning in various areas of medical image analysis has brought excellent performance gains. In particular, technologies based on deep learning in medical image registration can outperform traditional optimisation-based registration algorithms both in registration time and accuracy. Howev…
Improvements in the performance of deep neural networks have often come through the design of larger and more complex networks. As a result, fast memory is a significant limiting factor in our ability to improve network performance. One approach to overcoming this limit is the design of sparse neural networks, which ca…
Lottery tickets find good initializations for IMP with sparse training.
problem Finding good initializations for iterative magnitude pruning (IMP) in sparse networks.
method Empirical study of IMP performance with varying pre-training data and iterations.
result Training on a small fraction of data suffices to obtain good initializations for IMP.
As an important part of the power system, power load forecasting directly affects the national economy. The data shows that improving the load forecasting accuracy by 0.01% can save millions of dollars for the power industry. Therefore, improving the accuracy of power load forecasting has always been the pursuing goals…
Learning to approximate a separable function is hard, requiring many samples even with sparse networks.
problem Learning the separable function x↦∑i=1dxi2 with limited samples. method Sparse neural networks vs. dense neural networks, explicit regularization.
result The sample complexity for dense networks is O(d2.5) with explicit regularization, better than O(d4). Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we int…
This paper presents a new artificial neuron model capable of learning its receptive field in the topological domain of inputs. The model provides adaptive and differentiable local connectivity (plasticity) applicable to any domain. It requires no other tool than the backpropagation algorithm to learn its parameters whi…
Detection of dense cycles in graphs reveals a gap between easy detection and hard recovery.
problem Detecting and recovering dense cycles in Erdős-Rényi graphs.
method Characterization of computational thresholds for detection and recovery using low-degree polynomial algorithms.
result A gap exists between the detection and recovery thresholds for certain parameter regimes.
Inverse modeling for the estimation of non-Gaussian hydraulic conductivity fields in subsurface flow and solute transport models remains a challenging problem. This is mainly due to the non-Gaussian property, the non-linear physics, and the fact that many repeated evaluations of the forward model are often required. In…
DVNet efficiently segments large neurovascular datasets using skip connections.
problem Challenges in segmenting large neurovascular datasets from high-throughput microscopy data.
method A fully-convolutional, deep, and densely-connected encoder-decoder network with skip connections.
result DVNet achieves superior performance in semantic segmentation of neurovascular datasets.
From human crowds to cells in tissue, the detection and efficient tracking of multiple objects in dense configurations is an important and unsolved problem. In the past, limitations of image analysis have restricted studies of dense groups to tracking a single or subset of marked individuals, or to coarse-grained group…
The study finds a theoretical bound for pre-training iterations needed for pruning to yield good subnetwork performance.
problem Discovering efficient subnetworks within pre-trained dense networks.
method Mathematical analysis of a two-layer, fully-connected network, validating with a multi-layer perceptron trained on MNIST.
result A logarithmically dependent threshold on dataset size for successful pruning.
Sparse neural networks can match dense models on Lipschitz functions.
problem Sparse networks are more efficient but lack theoretical guarantees.
method Formal model of sparse networks, LSH-based routing function, Lipschitz function approximation.
result Sparse networks can approximate dense networks on Lipschitz functions.
While sparse inverse covariance matrices are very popular for modeling network connectivity, the value of the dense solution is often overlooked. In fact the L2-regularized solution has deep connections to a number of important applications to spectral graph theory, dimensionality reduction, and uncertainty quantificat…
Inducing sparseness while training neural networks has been shown to yield models with a lower memory footprint but similar effectiveness to dense models. However, sparseness is typically induced starting from a dense model, and thus this advantage does not hold during training. We propose techniques to enforce sparsen…
We solve 6-DoF localisation and 3D reconstruction using deep state-space models.
problem 6-DoF localisation and dense 3D reconstruction in spatial environments.
method Approximate Bayesian inference in a deep state-space model combining learning and domain knowledge.
result Near state-of-the-art performance on UAV flight data.
Dense neural networks learn efficiently with large datasets and noise.
problem Training neural networks with large, noisy datasets.
method Statistical mechanics and Monte Carlo simulations.
result Dense neural networks can handle large amounts of patterns and recognize patterns at high signal-to-noise ratios.
Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model. Recent reports (Han et al., 2015; Narang et al., 2017) prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction…
A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of mo…