Proposes a new theoretical framework for deep locally connected ReLU networks.
problem Understanding theoretical properties of deep locally connected networks.
method Teacher-student setting, explicit formulation of data distribution, disentangled representations, compatibility with regularization techniques.
result Theoretical framework facilitates analysis of practical issues like overfitting and generalization.
Wide neural networks have non-attracting local minima.
problem Understanding the impact of suboptimal local minima in deep and wide neural networks.
method Construction of non-attracting local minima and saddle points in wide neural networks.
result Wide neural networks have non-attracting local minima, even though they are not negatively impacted by suboptimal local optima.
Deep learning networks have connected sublevel sets, avoiding local minima.
problem Finding local minima in deep learning networks.
method Analyzing sublevel sets of loss functions in over-parameterized neural nets.
result Sublevel sets are connected and unbounded, ensuring all global minima are accessible.
The paper proves skip connections help neural networks avoid shallow local minima.
problem Understanding how skip connections affect the loss landscape of deep neural networks.
method Theoretical analysis of the topology of loss landscapes of deep ReLU neural networks with skip connections.
result Skip connections help control the connectedness of sub-level sets, avoiding shallow local minima.
Convolutional networks outperform shallow classifiers on certain tasks due to hierarchical structure.
problem Understanding why convolutional networks outperform shallow classifiers on specific tasks.
method Approximation theory, visual tasks with deterministic scrambling, and network performance evaluation.
result Hierarchical structure is crucial for convolutional networks' performance on certain tasks, but not all.
Deep ResNets can have better local minima than linear predictors.
problem Understanding the optimization landscape of deep ResNets compared to linear predictors.
method Analyzing the optimization landscape of ResNets with multiple residual blocks, showing geometric conditions under which ResNets have better local minima.
result Theorem showing that any critical point in the optimization landscape of deep ResNets is either at least as good as the best linear predictor or has a strictly negative eigenvalue in its Hessian.
This paper explores optimization methods for deep learning.
problem How to effectively train neural networks.
method Discusses gradient explosion/vanishing, initialization, normalization, SGD, adaptive gradient methods, distributed methods, and global issues.
result Provides theoretical insights and practical solutions for neural network training.
Deep networks trained with Hebbian updates perform similarly to back-propagation on image datasets.
problem Training deep networks with realistic asymmetric connections and updates.
method Use Hebbian updates with separate feedforward and feedback weights, and local rule for updates.
result Similar performance to back-propagation achieved with Hebbian updates on challenging image datasets.
While the optimization problem behind deep neural networks is highly non-convex, it is frequently observed in practice that training deep networks seems possible without getting stuck in suboptimal points. It has been argued that this is the case as all local minima are close to being globally optimal. We show that thi…
New method relaxes spatial invariance in locally connected layers, improving accuracy.
problem Improving classification accuracy with locally connected layers.
method Designing a low-rank locally connected layer with varying spatially varying combining weights.
result Relaxing spatial invariance improves classification accuracy over convolution and locally connected layers.
The backpropagation of error algorithm (BP) is impossible to implement in a real brain. The recent success of deep networks in machine learning and AI, however, has inspired proposals for understanding how the brain might learn across multiple layers, and hence how it might approximate BP. As of yet, none of these prop…
Shortcut connections in ResNet help avoid local optima, leading to efficient training.
problem Understanding why shortcut connections in ResNet lead to efficient training.
method Two-layer non-overlapping convolutional ResNet, gradient descent with proper normalization.
result Gradient descent avoids spurious local optima, converging to a global optimum.
The paper calculates bounds on the local Lipschitz constants of neural network layers.
problem Understanding the Lipschitz constants of neural network layers for robustness analysis.
method Analytical approach to determine upper bounds on local Lipschitz constants of affine-ReLU functions.
result The method produces tighter bounds than the standard conservative bound, especially for small perturbations.
Empirical study finds deep learning assumptions often incorrect.
problem Widespread assumptions about neural networks are often incorrect.
method Empirical evaluation of neural network assumptions using various techniques.
result Proves existence of suboptimal local minima and demonstrates practical implications.
Shallow networks with local learning rules can match deep learning performance.
problem Training deep neural networks is biologically implausible; the goal is to achieve similar performance with shallow networks.
method Investigated shallow networks with one hidden layer and a single readout layer, using various local learning rules for the hidden layer and supervised learning for the readout layer.
result Shallow networks can achieve test accuracy comparable to deep learning models, suggesting the use of different datasets for testing.
Proposes LC-ST-FCN for better ride-sourcing demand forecasting.
problem Local statistical differences in ride-sourcing demand across a city.
method LC-ST-FCN framework combining 3D and 2D convolutions, locally connected layers.
result Significant improvements in demand forecasting compared to baselines.
The aim of this work is to enable inference of deep networks that retain high accuracy for the least possible model complexity, with the latter deduced from the data during inference. To this end, we revisit deep networks that comprise competing linear units, as opposed to nonlinear units that do not entail any form of…
New deep architecture models uncertainty by sharing neural connectivity patterns.
problem Uncertainty modeling in deep neural networks remains challenging.
method Proposes a new deep architecture that shares neural connectivity patterns between generative and discriminative networks to model a confounder.
result Demonstrates significant improvement in uncertainty estimation compared to state-of-the-art methods.
Theoretical analysis of CNNs' inductive biases and their efficiency in approximating functions.
problem Understanding and optimizing the inductive biases in deep CNNs.
method Theoretical analysis combining multichanneling, downsampling, weight sharing, and locality.
result Deep CNNs with O ( log d ) \mathcal{O}(\log d) O ( log d ) depth can approximate any continuous function, and require O ~ ( log 2 d ) \widetilde{\mathcal{O}}(\log^2d) O ( log 2 d ) samples for sparse functions. A new method for deep clustering uses autoencoded embeddings and local manifold learning.
problem Improving clustering performance in deep learning models.
method Learning an autoencoded embedding, then clustering the underlying manifold using a shallow algorithm.
result UMAP is best at finding the most clusterable manifold in the embedding.
Proposes a deep neural network for multi-dimensional functional data classification.
problem Classifying multi-dimensional functional data with non-Gaussian distributions.
method Trains a deep neural network on the principle components of the training data.
result FDNN achieves minimax optimality when log density ratio has a locally connected modular structure.
Improved loss functions adapt to weight-space anisotropy, outperforming isotropic counterparts.
problem Adapting to the anisotropic nature of deep weight spaces for better performance.
method Refined local entropic loss functions restricted to a subset of weights, exploiting anisotropy.
result Partial local entropies outperform isotropic counterparts on image classification tasks.
Overparameterized autoencoders can memorize training examples.
problem Understanding generalization in overparameterized neural networks.
method Analyzing autoencoders of varying depths and types.
result Autoencoders concentrate learned functions around training examples.
Study analyzes error in ReLU networks with local connections.
problem Improving neural network performance and understanding approximation errors.
method Analyzed approximation error of ReLU networks with local connections.
result Error estimate depends on depth and width of hidden layers.
Expressive efficiency refers to the relation between two architectures A and B, whereby any function realized by B could be replicated by A, but there exists functions realized by A, which cannot be replicated by B unless its size grows significantly larger. For example, it is known that deep networks are exponentially…
Proposes neuron alignment to optimize mode connectivity in neural networks.
problem Understanding and optimizing mode connectivity in deep neural networks.
method Introduces neuron alignment to approximate optimal weight permutations and improve mode connectivity.
result Neuron alignment significantly alleviates robust loss barriers and improves model robustness and accuracy.
New metric measures deep learning robustness to noise.
problem Deep learning models are vulnerable to noise.
method Formal definition of robustness as localized Lipschitz constant.
result New metric evaluated on competitive vision datasets.
LP algorithm optimizes neural networks with architectural constraints.
problem Training neural networks with specific architectural constraints.
method Lagrangian optimization and saddle point search in adjoint space.
result LP algorithm is fully parallelizable and feasible for training deep networks.
Measures neural network decision boundary volume to predict model performance.
problem Understanding the geometry of deep learning models for better performance.
method Local surface volumes to measure decision boundary, applying Weyl's tube formula.
result Smaller surface volume correlates with higher classification accuracy.
Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently for sparse networks. Several methods exist for pruning connections of a neural network after it was trained without conne…
New theory for local parameterization of deep ReLU networks.
problem Determining local parameters of deep ReLU neural networks.
method Introducing local lifting operators and charts of a manifold, deriving necessary and sufficient conditions for local identifiability.
result Sharp and testable conditions for local identifiability of deep ReLU networks.
This paper improves deep neural network approximation for fully connected networks, achieving optimal convergence rates.
problem Improving approximation of fully connected deep neural networks for optimal convergence rates.
method Deriving approximation bounds specifically for a narrower fully connected deep neural network.
result Achieves an optimal rate (up to a logarithmic factor) for fully connected deep neural networks.
Paper proposes a deep learning strategy for efficient FC management.
problem Inaccurate and conservative FC dimensioning approaches in realistic settings.
method Uses a Convolutional Neural Network (CNN) to dynamically modulate FC resources.
result Achieves a maximum rejection rate of 3% and 37.5% resource savings.
Improved spectroscopy classification with deep learning and synthetic data.
problem Identifying chlorinated solvents in Raman spectra.
method Locally-connected neural network (NN) for binary classification, autoencoder-based outlier detection, and synthetic training data.
result The proposed method outperforms existing algorithms in accuracy and robustness.
Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to …
Tensor networks improve medical image classification performance.
problem Improving medical image classification accuracy.
method Extending tensor networks to medical image analysis, focusing on 2D images.
result Tensor networks achieve comparable performance to deep learning methods with fewer hyperparameters and resources.
Deep networks avoid bad local minima with no bad valleys.
problem Finding sub-optimal local minima in deep neural networks.
method Analyzing over-parameterized networks with specific activation and loss functions.
result No bad local valleys, implying no sub-optimal strict local minima.
The paper studies deep neural networks with Gaussian weights and finds their asymptotic behavior.
problem Understanding the behavior of deep neural networks with large width.
method Function-space perspective, Gaussian process analysis, weak convergence in large-width limit.
result Deep neural networks with large width converge to a continuous Gaussian process.
A new channel locality block improves CNN performance.
problem Improving the performance of convolutional neural networks.
method Proposed a variant of Squeeze-and-Excitation block using convolutional layers to learn nearby channel correlation.
result Our C-Local block achieved higher accuracy than the standard SE block on the cifar-10 dataset.
LocalDrop uses local Rademacher complexity for neural network regularization.
problem Overfitting in deep neural networks.
method Developed a new regularization function based on local Rademacher complexity.
result Demonstrated effectiveness of LocalDrop through extensive experiments.
This paper proves adding neurons eliminates all bad local minima in deep learning.
problem Eliminating all suboptimal local minima in deep learning models.
method Adding one special neuron per output unit eliminates all suboptimal local minima of deep neural networks.
result At every local minimum, the original neural network parameters are a global minimum.
Neural networks' optimization dynamics are confined to a single basin despite connected basins in the loss landscape.
problem Neural networks' optimization dynamics are confined to a single basin despite connected basins in the loss landscape.
method Identifying entropic barriers arising from the interplay between curvature variations along low-loss paths and noise in optimization dynamics.
result Curvature-induced entropic forces bias noisy dynamics back toward the endpoints, explaining the confinement and connectivity of solutions.
This paper simplifies deep ReLU networks into local linear models for better interpretability.
problem Limited transparency and interpretability of deep neural networks, especially ReLU networks.
method Local linear representation and equivalent set of local linear models (LLMs).
result Simplified deep ReLU networks for better interpretability and diagnostics.
Deep neural networks reveal a low-dimensional manifold structure in data.
problem Understanding the structure of data for better model performance.
method Model-centric analysis of the data manifold using the local data matrix and Fisher information matrix.
result The dataset lies on a data leaf with a dimension bounded by the number of labels.
Deep networks with a wide layer ensure sublevel set connectivity.
problem Ensuring connectivity of sublevel sets in deep learning.
method Analyzing the connectivity of sublevel sets in deep neural networks with a specific layer width.
result A single wide layer of width N + 1 N+1 N + 1 suffices to prove connectivity of sublevel sets. New framework tackles deep learning issues like local traps and miscalibration.
problem Local traps and miscalibration in deep neural networks.
method Sparse deep learning framework with prior annealing algorithms.
result Proposed method successfully addresses local traps and miscalibration.
SGD converges with positive probability for non-convex deep neural networks under specific conditions.
problem Convergence of SGD for non-convex deep neural networks.
method Established local convergence with positive probability under local Łojasiewicz condition and additional structural assumption.
result SGD converges with positive probability for non-convex deep neural networks under specific conditions.
Deep linear networks avoid spurious local minima under certain conditions.
problem Existence of spurious local minima in deep linear networks.
method Reduction to two-layer case, quadratic loss analysis, and perturbation argument to show full rank property.
result Deep linear networks have no spurious local minima under specific conditions.