New loss functions reveal layer roles in deep neural networks.
problem Understanding the role of individual layers in deep neural networks.
method Derived Deep Gaussian Layer-wise loss functions (DGLs) using Gaussian Processes and SGD.
result First explicit and competitive layer-wise loss functions for deep neural networks.
Proposes DT-LET for better transfer learning between domains with different resolutions.
problem Transfer learning between domains with different resolutions often fails due to mismatched hidden layers.
method Develops DT-LET model to find the best matching layers for transfer based on high-level feature correspondence.
result DT-LET achieves superior results in cross-domain recognition/classification tasks, validating the necessity of layer correspondence searching.
In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …
EigenGAN discovers interpretable dimensions in GAN layers for semantic control.
problem Lack of explicit dimensions to control semantic attributes in GAN layers.
method EigenGAN embeds linear subspaces with orthogonal bases into each generator layer, learning eigen-dimensions corresponding to semantic attributes via adversarial training.
result EigenGAN can produce samples with continuous changes corresponding to specific semantic attributes.
ReLU networks trained with MILPs match deep learning accuracy.
problem Training deep neural networks efficiently.
method Iterative training with Mixed Integer Linear Programs (MILPs).
result ReLU networks can be trained with MILPs achieving similar accuracy to deep learning methods.
We present convolutional neural network (CNN) based approaches for unsupervised multimodal subspace clustering. The proposed framework consists of three main stages - multimodal encoder, self-expressive layer, and multimodal decoder. The encoder takes multimodal data as input and fuses them to a latent space representa…
New layer sparsity concept improves neural networks.
problem Improving neural network efficiency and interpretability.
method Formulated layer sparsity, introduced regularization and refitting schemes.
result Generated more compact and accurate neural networks.
DeepAM optimizes deep neural networks for image super-resolution.
problem Image super-resolution using deep neural networks.
method L-layer analysis dictionary model with IPAD and CAD.
result DeepAM outperforms deep neural networks with back-propagation.
We propose here a multiplex network approach to investigate simultaneously different types of dependency in complex data sets. In particular, we consider multiplex networks made of four layers corresponding respectively to linear, non-linear, tail, and partial correlations among a set of financial time series. We const…
Lookahead pruning extends single-layer optimization to multi-layer, outperforming magnitude-based pruning.
problem Pruning neural networks to reduce computational cost and memory usage.
method Developed a multi-layer optimization approach extending the single-layer optimization of magnitude-based pruning.
result Consistently outperforms magnitude-based pruning on various networks, especially in high sparsity.
The paper shows how random ReLU networks converge to smooth splines.
problem Understanding the behavior of shallow ReLU neural networks with random weights.
method Mathematical analysis of L2-regularized regression and gradient descent.
result Random ReLU networks converge to smooth splines as the number of hidden nodes increases.
Graph U-Nets improve graph representation learning.
problem Challenges in learning representations for graph data.
method Proposed novel graph pooling and unpooling operations in a graph U-Nets architecture.
result Graph U-Nets achieve better performance on node and graph classification tasks.
Develops a novel global pooling framework using optimal transport.
problem Sub-optimal performance in global pooling operations.
method Regularized Optimal Transport (ROT) for generalized global pooling.
result ROTP layers can improve performance in various machine learning scenarios.
Bayesian CNNs with many channels are equivalent to Gaussian processes.
problem Understanding the behavior of deep convolutional networks in the infinite channel limit.
method Deriving an equivalence between multi-layer convolutional neural networks and Gaussian processes, introducing a Monte Carlo method for estimation.
result The GPs corresponding to CNNs with and without weight sharing are identical in the infinite channel limit.
Introduces Causal Energy Minimization to understand Transformer layers.
problem Empirical parameterization of Transformer blocks remains largely unexplored.
method Causal Energy Minimization framework that recasts Transformer layers as optimization steps on conditional energy functions.
result Identifies design space for Transformer layers including weight sharing and energy-based interpretations.
Establishes connection between MTDNN and multitask GP, revealing weight correlation as key to task sharing.
problem Limited theoretical understanding of information sharing in MTDNN.
method Derives multitask GP kernels for MTDNN and MTBNN, showing shared hyper-parameters and last layer weights.
result Information sharing in MTDNN is due to weight correlation, not intermediate layer weights.
Layer normalization with activations prevents Gram matrix rank collapse at initialization.
problem Rank collapse in Gram matrices at initialization slows training in deep networks.
method Proved that layer normalization, with activation layers, biases Gram matrix towards identity matrix at exponential rate.
result Layer normalization with activations biases Gram matrix towards identity matrix at exponential rate with depth at initialization.
We introduce backdrop, a flexible and simple-to-implement method, intuitively described as dropout acting only along the backpropagation pipeline. Backdrop is implemented via one or more masking layers which are inserted at specific points along the network. Each backdrop masking layer acts as the identity in the forwa…
In this paper we propose a function space approach to Representation Learning and the analysis of the representation layers in deep learning architectures. We show how to compute a weak-type Besov smoothness index that quantifies the geometry of the clustering in the feature space. This approach was already applied suc…
An interesting approach to analyzing neural networks that has received renewed attention is to examine the equivalent kernel of the neural network. This is based on the fact that a fully connected feedforward network with one hidden layer, a certain weight distribution, an activation function, and an infinite number of…
We consider the problem of reconstructing a signal from multi-layered (possibly) non-linear measurements. Using non-rigorous but standard methods from statistical physics we present the Multi-Layer Approximate Message Passing (ML-AMP) algorithm for computing marginal probabilities of the corresponding estimation proble…
The introduction of convolutional layers greatly advanced the performance of neural networks on image tasks due to innately capturing a way of encoding and learning translation-invariant operations, matching one of the underlying symmetries of the image domain. In comparison, there are a number of problems in which the…
We investigate the problem of factorizing a matrix into several sparse matrices and propose an algorithm for this under randomness and sparsity assumptions. This problem can be viewed as a simplification of the deep learning problem where finding a factorization corresponds to finding edges in different layers and valu…
Recently, the deep-belief-networks (DBN) based voice activity detection (VAD) has been proposed. It is powerful in fusing the advantages of multiple features, and achieves the state-of-the-art performance. However, the deep layers of the DBN-based VAD do not show an apparent superiority to the shallower layers. In this…
Algorithm predicts performance of learning in multi-layer networks with matrix-valued hidden variables.
problem Signal recovery and learning in multi-layer neural networks with matrix-valued hidden variables.
method Unified approximation algorithm for MAP and MMSE inference, extending ML-VAMP to handle matrix-valued unknowns.
result Performance of ML-Mat-VAMP algorithm can be predicted in a random large-system limit.
New method bounds singular values of convolutional kernels to stabilize gradients.
problem Stable gradients in convolutional neural networks.
method Frobenius norm regularization for convolutional kernels.
result Bounded singular values of transformation matrices.
Efficient memory layer improves graph neural networks for graph classification and regression.
problem Efficiently learning node representations and graph coarsening for arbitrary graph topology.
method Introduces a memory layer for GNNs that learns node representations and graph coarsening, and two new networks: MemGNN and GMN.
result Proposed models achieve state-of-the-art results in graph classification and regression benchmarks.
Improved recommendation systems using multi-layer embeddings reduce model size while maintaining accuracy.
problem Improving model accuracy in recommendation systems while minimizing model size.
method Introducing a multi-layer embedding training (MLET) architecture that trains embeddings via a sequence of linear layers.
result Substantial advantages in model accuracy and memory footprint are achieved with reduced embedding dimensions.
This study reveals a Min-Max property in LeNet's convolutional layers, enhancing adversarial robustness.
problem Uncertainty in the connection weights of convolutional layers in neural networks.
method Demonstrates the Min-Max property through back propagation-based training and a simplified convolution formulation.
result The Min-Max property improves adversarial robustness, indicating a stronger uncertainty in the model parameters.
Wavelets help compress neural networks efficiently.
problem Efficiently compressing linear layers in neural networks.
method Learnable wavelet transforms to compress RNNs.
result Wavelet compressed RNNs have fewer parameters and perform competitively.
This paper introduces a new method for neural networks that doesn't need a global coordinate system.
problem The lack of a global coordinate system in neural networks limits their performance and explainability.
method Proposes a learnable topological layer that works in a general metric space (Hilbert space) without requiring a Euclidean space.
result The proposed method eliminates the need for a costly parametrization stage and achieves optimal network performance.
Study compares investor networks and cellphone communication networks to analyze layer structures.
problem Comparing investor networks and cellphone communication networks to understand information diffusion.
method Used clustering algorithms to detect layer structures in ego networks of both networks.
result Nodes in both networks can be divided into two groups with specific layer structures.
It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of eva…
Deep ReLU networks can be simplified to a three-layer model.
problem Understanding the behavior of deep neural networks.
method Constructive proof and algorithm to transform deep networks into shallow ones.
result Deep ReLU networks can be represented by a simpler three-layer structure.
Wide neural networks converge to Gaussian processes, improving generalization.
problem Understanding the generalization of wide neural networks, especially deep equilibrium models.
method Investigation of deep equilibrium models (DEQs) with infinite-depth layers, focusing on their convergence to Gaussian processes as width and depth approach infinity.
result Wide DEQs converge to Gaussian processes, maintaining generalization performance.
Proposes FROT for high-dimensional data, avoiding curse of dimensionality.
problem High-dimensional data challenges in optimal transport.
method Feature selection and min-max optimization for robust transport plan.
result FROT achieves state-of-the-art performance in semantic correspondence.
This work defines a new function space for multi-layer neural networks.
problem Characterizing the function space of multi-layer neural networks.
method Defining a neural Hilbert ladder (NHL) as an infinite union of reproducing kernel Hilbert spaces (RKHSs).
result Established theoretical properties of the new function space, including generalization guarantees and dynamics of random fields.
CNNs encode data augmentation transformations, especially in early layers.
problem Whether neural network features encode data augmentation transformations.
method Systematic approach using pre-trained vision models to predict augmentation transformations.
result Neural network features encode data augmentation transformations, especially in early layers.
We present a novel architecture, the "stacked what-where auto-encoders" (SWWAE), which integrates discriminative and generative pathways and provides a unified approach to supervised, semi-supervised and unsupervised learning without relying on sampling during training. An instantiation of SWWAE uses a convolutional ne…
Enhanced CNN kernels improve image classification accuracy.
problem Improving classification accuracy of CNNs.
method Local Average Pooling and random image patch representation.
result 89% accuracy on CIFAR-10, matching AlexNet performance.
A framework for transformer attention layers derived from SVR.
problem Developing principled attention mechanisms for transformers.
method Mapping self-attention to SVR, deriving new attention types.
result Improved transformer performance and efficiency.
Hierarchical nucleation patterns emerge in deep neural network layers.
problem Understanding the generation of meaningful representations in deep neural networks.
method Analysis of the probability density of ImageNet dataset across hidden layers.
result Density peaks in subsequent layers mirror the semantic hierarchy of concepts, resembling nucleation process.
New method compresses deep learning layers using tensor decomposition.
problem Reduction of computation cost and interpretability for tensor data.
method CP-decomposition to compress convolutional layers in deep learning.
result Reduces model complexity and maintains prediction performance.
Spotlight method finds hidden errors in deep learning models.
problem Systematic errors in deep learning models on rare data subsets.
method Shining a spotlight on hidden layer representations to find poor performance areas.
result Identifies semantically meaningful areas of weakness in various models.
Study clarifies Bayesian generalization error in CBM for 3-layered linear neural networks.
problem Understanding the generalization error in concept bottleneck models.
method Mathematical analysis of Bayesian generalization error and free energy in CBM for 3-layered linear neural networks.
result CBM significantly alters the parameter region and Bayesian generalization error compared to standard models.
This paper introduces a hierarchical associative memory model with multiple layers.
problem Limitations of traditional associative memory models with only one hidden layer.
method Develops a fully recurrent model with arbitrary layers, including locally connected ones, and a corresponding energy function.
result The model can dynamically assemble memories using weights from lower layers and higher layers' rules.
SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.
problem Sparse autoencoders' reconstruction error varies across layers, not fitting existing scaling laws.
method Cross-layer study of 844 SAE checkpoints, fitting and regressing on manifold geometry.
result Manifold geometry predicts layer-dependent width exponents in SAEs, with transferable coefficients.
LLEB uses a learned prior to quantify neural network uncertainty.
problem Quantifying uncertainty in neural network predictions.
method LLEB uses a learnable prior as a normalizing flow to maximize the evidence lower bound.
result LLEB performs on par with existing approaches in uncertainty quantification.