CDPL-Net integrates CNN and DL for better image representation.
problem Improving image representation learning by combining CNN and DL.
method CDPL-Net architecture combining CNN and DPL layers, using l1-norm for sparse representation, and efficient stochastic gradient descent.
result Enhanced performance compared to state-of-the-art methods.
A new metric measures saturation of neural network layers.
problem Analyzing the quality of latent representations in neural networks.
method Layer Saturation metric based on spectral analysis.
result Saturation is related to generalization and predictive performance.
Study shows how deep network representations can be transferred between datasets and tasks.
problem Transferability of deep network representations across datasets and tasks.
method Examined layer-wise transferability of representations in deep networks across multiple datasets and tasks.
result Interesting empirical observations on layer-wise transferability of representations.
Lean 2-layer RBMs achieve similar representational power as single-layer RBMs with fewer parameters.
problem Understanding and quantifying the representational power of multi-layer RBMs.
method Inherent Structure Capacity (ISC) and Lean RBMs.
result 2-layer RBMs can achieve the same representational power as single-layer RBMs with fewer parameters.
New loss functions reveal layer roles in deep neural networks.
problem Understanding the role of individual layers in deep neural networks.
method Derived Deep Gaussian Layer-wise loss functions (DGLs) using Gaussian Processes and SGD.
result First explicit and competitive layer-wise loss functions for deep neural networks.
Proposes a method to determine the most compact representation in deep neural networks.
problem Finding the most compact representation in deep neural networks.
method Layer-wise pruning method considering layer importance and total target sparsity.
result Achieved up to 75% better top-5 accuracy with 75% compression than baseline.
Multi-layered representation is believed to be the key ingredient of deep neural networks especially in cognitive tasks like computer vision. While non-differentiable models such as gradient boosting decision trees (GBDTs) are the dominant methods for modeling discrete or tabular data, they are hard to incorporate with…
In this paper we propose a function space approach to Representation Learning and the analysis of the representation layers in deep learning architectures. We show how to compute a weak-type Besov smoothness index that quantifies the geometry of the clustering in the feature space. This approach was already applied suc…
Self-supervised and supervised methods learn similar intermediate visual representations but diverge in final layers.
problem Comparing self-supervised and supervised methods for visual learning.
method Comparison of contrastive self-supervised and supervised methods on simple image data.
result Contrastive and supervised methods learn similar intermediate representations but diverge in final layers.
Modified RV-coefficient reveals how training affects neural network representations.
problem Understanding how training affects intermediate representations in convolutional neural networks.
method Experimented with modified RV-coefficient (RV2) to compare activation patterns in deep networks trained on varying amounts of data and layers.
result RV2 successfully recovered expected similarity patterns and provided interpretable similarity matrices.
Deep networks learn low-dimensional yet complex data representations.
problem Understanding the intrinsic dimensionality of deep neural network representations.
method Analysis of intrinsic dimensionality across multiple layers of trained networks.
result The intrinsic dimensionality of data representations in deep networks is significantly lower than the number of units in each layer.
Task loss matching misrepresents similarity between neural network layers.
problem Measuring similarity between neural network layers using task loss matching.
method Task loss matching vs. direct matching; comparison with CCA and CKA.
result Direct matching provides a better similarity index than task loss matching.
Neural networks learn general representations for solving differential equations.
problem Solving parametrized boundary value problems using neural networks.
method Singular Vector Canonical Correlation Analysis (SVCCA) for measuring generality.
result First hidden layer learns general representations, deeper layers become more specific.
Revisits the connection between neural networks and the Kolmogorov-Arnold theorem.
problem Explains the limitations of using the Kolmogorov-Arnold theorem to explain neural networks with multiple hidden layers.
method Derives modifications of the Kolmogorov-Arnold representation that transfer smoothness properties to the outer function and can be well approximated by ReLU networks.
result Shows that a deep neural network with most layers approximating the interior function is a more natural interpretation of the Kolmogorov-Arnold representation.
Two-layer neural networks can approximate functions with fractal singularities.
problem Characterizing functions that can be represented by infinitely wide two-layer neural networks.
method Representation formulas and pointwise properties analysis.
result Functions with fractal or curved singularities cannot be represented by two-layer networks with finite path-norm.
Researchers analyze the geometric and statistical properties of transformer model representations.
problem Understanding the semantic structure of large transformer models across various data types.
method Characterization of geometric and statistical properties through analysis of intrinsic dimension and neighbor composition.
result The semantic information of the dataset is better expressed at the end of the first peak in transformer models.
Deep learning improves sparse representation for better classification.
problem Improving classification accuracy using sparse representation.
method A transductive deep learning network combining convolutional autoencoder and fully-connected layers.
result The proposed network achieves better classification results than state-of-the-art SRC methods.
Kernel and neural embeddings improve optimization and generalization in deep networks.
problem Improving optimization and generalization in deep neural networks.
method Investigated three kernel representations and their neural network approximations, comparing their optimization and generalization properties.
result Kernel and neural embeddings enhance both optimization and generalization in deep networks.
Researchers create integral representations for two-layer ReLU networks with quantitative bounds.
problem Approximating functions with two-layer ReLU networks using explicit integral representations.
method Developed integral representations involving harmonic extension and projection, providing L2 bounds. result Functions can be approximated with L2 errors independent of dimension or degree, depending on coefficients and distribution. The paper explores how AI trading agents' similar information representation can cause financial market instability.
problem Systemic instability in AI-dominated financial markets due to similar information representation.
method Structural multi-agent market model with two-layer decision architecture for AI agents.
result Representation homogeneity can lead to systemic instability in financial markets.
Paper proposes a deep subspace clustering method using multi-level representations.
problem Deep subspace clustering of images.
method Convolutional autoencoders with multiple fully-connected layers for multi-level representations, loss minimization with iterative updates.
result The method outperforms state-of-the-art methods on real-world datasets.
The Soft Nearest Neighbor Loss improves representation quality and generalization.
problem Improving the quality of representations in machine learning models.
method Explored and expanded the Soft Nearest Neighbor Loss to measure class entanglement.
result Maximizing the entanglement of different class representations leads to better generalization and uncertainty estimation.
The interbank market has a natural multiplex network representation. We employ a unique database of supervisory reports of Italian banks to the Banca d'Italia that includes all bilateral exposures broken down by maturity and by the secured and unsecured nature of the contract. We find that layers have different topolog…
Chaos in cerebellar cells enhances complexity of neural patterns.
problem Understanding how cerebellar granular layer represents complex information.
method Constructed a model of cerebellar granular layer with gap junctions, evaluated using reservoir computing.
result Chaotic dynamics in the cerebellar granular layer produce complex and diverse output patterns.
Subspace match fails to accurately assess neural network representations.
problem Understanding the learned representations of neural networks.
method Subspace match method to assess representation similarity.
result Representations with low subspace match can still be isomorphic.
Efficient memory layer improves graph neural networks for graph classification and regression.
problem Efficiently learning node representations and graph coarsening for arbitrary graph topology.
method Introduces a memory layer for GNNs that learns node representations and graph coarsening, and two new networks: MemGNN and GMN.
result Proposed models achieve state-of-the-art results in graph classification and regression benchmarks.
We study how finite Bayesian neural networks adapt their hidden representations.
problem Understanding how finite Bayesian neural networks differ from infinite ones.
method We analyze the asymptotics of learned feature kernels for various network architectures.
result The leading finite-width corrections to feature kernels have a universal form.
This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
Binary representation is desirable for its memory efficiency, computation speed and robustness. In this paper, we propose adjustable bounded rectifiers to learn binary representations for deep neural networks. While hard constraining representations across layers to be binary makes training unreasonably difficult, we s…
iREPA shows spatial structure, not global semantic, drives generation performance in REPA.
problem Understanding what aspect of the target representation matters for generation.
method Empirical analysis of 27 vision encoders, two modifications to REPA.
result Spatial structure, not global semantic, drives generation performance.
A new method for disentangled representations without supervision.
problem Learning disentangled representations in unsupervised learning.
method Constr-DRKM, a deep kernel method with orthogonality constraints.
result Constr-DRKM performs similarly to β-VAE on disentanglement metrics.
Combines multi-layer graphs to infer global network structure.
problem Leveraging domain knowledge in structure inference for multi-layer graphs.
method Mask combination of multi-layer graphs using optimization.
result Enhanced structure inference through multi-layer graph integration.
Defines metrics to compare neural network representations.
problem Comparing neural network representations across different architectures and tasks.
method Developed a family of metric spaces and modified existing measures to quantify representational dissimilarity.
result Identified relationships between neural representations and anatomical features.
Deep learning explained through spectral filtering of hierarchical features.
problem Understanding how deep neural networks learn useful representations from data.
method Neural Low-Degree Filtering (Neural LoFi) as a stylized limit of gradient-based training.
result Predicts how representations are selected layer by layer and explains emergence of concepts.
Wide neural networks' last hidden layers split into groups of redundant neurons.
problem Understanding why wide neural networks generalize well despite overfitting.
method Analyzed the last hidden layer representations of various convolutional neural networks.
result Wide hidden layers split into groups of redundant neurons, which help generalize.
Neural network learns manifold structure for semi-supervised learning.
problem Semi-supervised learning on manifolds with explicit graph representations.
method Feed-forward neural network with two layers, learning manifold structure and classification.
result Neural network achieves semi-supervised learning on non-trivial manifolds.
In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …
New layers estimate complex time-frequency masks without phase wrapping issues.
problem Lack of phase estimation in deep learning-based speech enhancement and source separation.
method Proposes magbook, phasebook, and combook layers for complex mask estimation.
result Match state-of-the-art performance on speaker separation datasets.
This paper investigates how forgetting affects neural network representations and stabilizes deeper layers.
problem Catastrophic forgetting in machine learning models trained on sequential tasks.
method Representational analysis techniques and empirical studies on CIFAR-10 and CIFAR-100 datasets.
result Deeper layers are disproportionately the source of forgetting, and methods to mitigate forgetting stabilize these layers.
A decoder helps interpret neural network layers.
problem Understanding the output of neural network layers.
method Developed a Classifier-Decoder (ClaDec) architecture.
result Reconstructed images from ClaDec are more relevant for classification.
AL2 progressively penalizes network activations to prevent overfitting.
problem Avoiding overfitting in neural networks with limited training data.
method Progressive Activation Loss (PAL) method to regularize network representations.
result AL2 outperforms traditional regularization methods on benchmark datasets.
Deep Convolutional Neural Networks (CNN) enforces supervised information only at the output layer, and hidden layers are trained by back propagating the prediction error from the output layer without explicit supervision. We propose a supervised feature learning approach, Label Consistent Neural Network, which enforces…
Kernel methods are powerful tools to capture nonlinear patterns behind data. They implicitly learn high (even infinite) dimensional nonlinear features in the Reproducing Kernel Hilbert Space (RKHS) while making the computation tractable by leveraging the kernel trick. Classic kernel methods learn a single layer of nonl…
DS2CF-Net learns hierarchical representations with deep coupled factorization and enriched prior.
problem Learning deep hierarchical representations from data.
method Dual-constrained Deep Semi-Supervised Coupled Factorization Network (DS2CF-Net) with enriched prior.
result DS2CF-Net achieves state-of-the-art performance in representation learning and clustering.
Enhances image captioning with novel context combination methods.
problem Improving machine learning for image captioning with structured learning and meaningful interpretation.
method Combines Feature Distribution Composition (FDC), Multiple Role Representation Crossover (MRRC) attention layers, and language decoder.
result Significantly improved image captioning performance (35.3%) and established new standards.
Group-equivariant subsampling layers improve CNNs' equivariance.
problem Non-translation equivariance in subsampling operations.
method Translation and group-equivariant subsampling/upsampling layers.
result Group-equivariant autoencoders learn equivariant representations.
Model stitching compares neural representations, revealing insights not captured by CKA.
problem Understanding internal neural representations.
method Model stitching connects neural network layers to study representations.
result Good networks trained differently can be stitched without performance drop.
Self-supervised learning improves by predicting known information, reducing labeled data needs.
problem Efficiently learn useful semantic representations without labeled data.
method Develops a mechanism exploiting statistical connections between pretext tasks to learn representations that solve downstream tasks.
result Proves linear layer yields small approximation error and drastically reduces labeled sample complexity.