Study on identifiability of deep polynomial neural networks.
problem Understanding when polynomial neural networks can be uniquely identified.
method Comprehensive analysis including various architectures, using tensor decompositions and Kruskal-type theorems.
result Identifiability conditions for deep PNNs, including layer width and activation degree constraints.
Deep Discrete Encoders (DDEs) tackle interpretable generative models for rich data with discrete latent layers.
problem Overparametrized, non-identifiable, and uninterpretable deep generative models in high-stakes applications.
method Directed graphical model with multiple binary latent layers, transparent identifiability conditions, scalable estimation pipeline.
result Transparent identifiability conditions and scalable estimation pipeline for interpretable DDEs.
In this paper, we tackle the problem of explanations in a deep-learning based model for recommendations by leveraging the technique of layer-wise relevance propagation. We use a Deep Convolutional Neural Network to extract relevant features from the input images before identifying similarity between the images in featu…
Paper uses SSC for identifying layers with identical community structures in DIMPLE networks.
problem Identifying layers with identical community structures in DIMPLE networks.
method Sparse Subspace Clustering (SSC) for identifying groups of layers with identical community structures.
result SSC leads to strongly consistent between-layer clustering under mild conditions.
Transformers without skip connections collapse token representations to a single direction.
problem Rapid convergence of token representations to a single direction in self-attention-only Transformers.
method Analysis of layer normalization, residual connections, and multi-head attention mechanisms.
result Residual connections prevent rank collapse in real Transformers, while MLPs generate new feature directions.
Defines metrics to compare neural network representations.
problem Comparing neural network representations across different architectures and tasks.
method Developed a family of metric spaces and modified existing measures to quantify representational dissimilarity.
result Identified relationships between neural representations and anatomical features.
XAI identifies key time steps for early crop classification.
problem Early crop classification with high accuracy and timeliness.
method Training a baseline model with LRP to identify important time steps.
result Identified a 21st April 2019 to 9th August 2019 timeframe with 0.75% accuracy loss.
Currently, the diagnosis of Autism Spectrum Disorder (ASD) is dependent upon a subjective, time-consuming evaluation of behavioral tests by an expert clinician. Non-invasive functional MRI (fMRI) characterizes brain connectivity and may be used to inform diagnoses and democratize medicine. However, successful construct…
Study examines infinite limits of transformer dynamics, identifying key parameterizations.
problem Understanding the training dynamics of transformer models in the feature learning regime.
method Analysis of infinite scaling limits using dynamical mean field theory.
result Identified parameterizations that admit well-defined infinite width and depth limits.
Two-layer networks favor simple features, especially in complex datasets.
problem Simplicity bias in neural networks over-reliing on simple features.
method Characterization of two-layer neural networks with small weights and gradient flow.
result Features learned in middle training stages are more useful for out-of-distribution transfer.
Residual Networks with convolutional layers are widely used in the field of machine learning. Since they effectively extract features from input data by stacking multiple layers, they can achieve high accuracy in many applications. However, the stacking of many layers raises their computation costs. To address this pro…
We introduce a novel description of the dynamics of the order book of financial markets as that of an effective colloidal Brownian particle embedded in fluid particles. The analysis of a comprehensive market data enables us to identify all motions of the fluid particles. Correlations between the motions of the Brownian…
ALMA improves clustering of multilayer networks.
problem Clustering multilayer networks with distinct layers and communities.
method Alternating minimization algorithm (ALMA) for simultaneous layer partition and community estimation.
result ALMA achieves higher accuracy than TWIST in clustering multilayer networks.
Study identifies stable configurations of intertwined threads with repulsive interactions.
problem Stable configurations of entangled systems with repulsive interactions.
method Analysis of steepest descent flow of an energy functional.
result Existence and uniqueness of stable configuration of two layers drifting apart at t1/3 rate. Study shows LLMs can remove half of layers without significant performance drop.
problem Understanding knowledge storage in LLMs' weights.
method Layer pruning and finetuning to identify and remove unnecessary parameters.
result Minimal degradation of performance after removing up to half of layers.
Two-layer CNNs can overfit well if initialized correctly.
problem Understanding the conditions for benign overfitting in over-parameterized CNNs.
method Extending analysis to fully trainable two-layer CNNs, examining initialization scaling effects.
result Initialization scaling of the output layer is crucial; large scales lead to fixed output behavior, small scales to complex interactions.
Proposes a co-hub node model for multiview graph learning.
problem Identifying shared graphical structures in heterogeneous datasets.
method Enforces structured sparsity on co-hub nodes across multiple views.
result Demonstrates improved precision and interpretive insight in learning multiview graphs.
Introduces Causal Energy Minimization to understand Transformer layers.
problem Empirical parameterization of Transformer blocks remains largely unexplored.
method Causal Energy Minimization framework that recasts Transformer layers as optimization steps on conditional energy functions.
result Identifies design space for Transformer layers including weight sharing and energy-based interpretations.
Paper analyzes why deeper layers of ViTs perform worse on out-of-distribution tasks.
problem Performance degradation of intermediate layers in ViTs under distribution shift.
method Extensive linear probing experiments across various benchmarks and fine-grained analysis of transformer modules.
result Probing feedforward network activations yields best performance under significant distribution shift.
The paper uses VGG-19 for plant species classification from leaf images.
problem Manual inspection of plant species by botanists is time-consuming.
method Transfer learning with VGG-19 for feature extraction and classification.
result The model achieves 99.70% accuracy in predicting plant species.
PR-GNN identifies salient brain regions for ASD biomarkers.
problem Identifying brain regions associated with neurological disorders.
method Pooling Regularized Graph Neural Network (PR-GNN) with novel salient region selection.
result PR-GNN outperforms baseline methods in ASD classification accuracy.
This paper explains double descent in linear neural networks, identifying new factors.
problem Understanding double descent in linear neural networks.
method Gradient flow derivation and necessary conditions for double descent.
result Singular values of input-output covariance matrix are important for double descent in two-layer models.
The study compares feed-forward and attention layers in language models.
problem Understanding the role of feed-forward and attention layers in language models.
method Empirical and theoretical analysis in a synthetic setting.
result Feed-forward layers learn simple distributional associations, while attention layers focus on in-context reasoning.
Optimizes CNN architectures by analyzing receptive fields without training.
problem CNNs often have too many layers, making them resource-intensive and less effective.
method Layer-wise analysis of receptive fields to identify unproductive layers.
result Identifies and removes unproductive layers, improving CNN performance and efficiency.
Proposes QEP to mitigate quantization error propagation in layer-wise post-training quantization.
problem Growth of quantization errors across layers degrades performance, especially in low-bit regimes.
method Quantization Error Propagation (QEP) framework that explicitly propagates and compensates for quantization errors.
result QEP-enhanced layer-wise PTQ achieves substantially higher accuracy, especially in low-bit regimes.
New mechanism discovered for feature learning in CNNs.
problem Understanding how CNNs learn features from images.
method Proposed Convolutional Neural Feature Ansatz linking filter covariances to patch-based AGOPs.
result Deep ConvRFM algorithm learns features similar to deep CNNs, improving performance.
AIM models explain deep learning in attention layers, offering solvable insights.
problem Understanding how deep learning models learn in attention layers.
method Statistical mechanics and random matrix theory.
result Closed-form predictions for Bayes-optimal generalization error and gradient descent performance.
Multilayer networks are a useful data structure for simultaneously capturing multiple types of relationships between a set of nodes. In such networks, each relational definition gives rise to a layer. While each layer provides its own set of information, community structure across layers can be collectively utilized to…
Channel Pruning, widely used for accelerating Convolutional Neural Networks, is an NP-hard problem due to the inter-layer dependency of channel redundancy. Existing methods generally ignored the above dependency for computation simplicity. To solve the problem, under the Bayesian framework, we here propose a layer-wise…
Interprets AI model for identifying boosted H → b̄b jets.
problem Difficulty in explaining AI model decisions due to complexity.
method Exploring Interaction Network (IN) model and Neural Activation Pattern (NAP) diagrams.
result NAP diagrams reveal important information about hidden layers' activity.
Identifies interpretable generative model for multivariate data.
problem Black-box architectures of deep generative models are often unidentified and difficult to interpret.
method Introduces Deep Discrete Encoder (DDE) Copula, a hierarchical binary latent variable model inside a copula framework.
result Establishes conditions for identification of DDE copula parameters and proves posterior consistency.
Overparametrized neural networks retain significant epistemic uncertainty even with sufficient data.
problem Epistemic uncertainty in overparametrized neural networks persists despite model identifiability.
method Analysis of non-identifiability and characterization of residual uncertainty in one-hidden-layer ReLU networks.
result Substantial parameter uncertainty remains even when the underlying function is fully identified.
Various forms of representations may arise in the many layers embedded in deep neural networks (DNNs). Of these, where can we find the most compact representation? We propose to use a pruning framework to answer this question: How compact can each layer be compressed, without losing performance? Most of the existing DN…
Simple k-NN filtering improves model accuracy on noisy labels.
problem Training models with noisy labels reduces performance and is hard to identify.
method A simple k-nearest neighbor-based filtering approach on the logit layer. result Improves model accuracy compared to recent methods.
New framework identifies hidden risks and optionality in American options.
problem Underestimation of flexibility and convexity in early-exercise features.
method Introducing stochasticity into underlying determinants to quantify hidden risks and optionality.
result Remedies conventional pricing systems that underestimate optionality.
FPGAs have become a popular choice for deploying deep learning architectures (DLA). There are many researchers that have explored the deployment and mapping of DLA on FPGA. However, there has been a growing need to do design-time hardware-software co-verification of these deployments. To the best of our knowledge this …
Layer normalization improves federated learning with skewed labels.
problem Label skewness in federated learning datasets.
method Identified feature normalization as key mechanism; applied to latent features before classifier.
result Normalization accelerates global training and improves convergence under extreme label shift.
Study clarifies Bayesian generalization error in CBM for 3-layered linear neural networks.
problem Understanding the generalization error in concept bottleneck models.
method Mathematical analysis of Bayesian generalization error and free energy in CBM for 3-layered linear neural networks.
result CBM significantly alters the parameter region and Bayesian generalization error compared to standard models.
New method uses statistical physics to detect financial market manipulation.
problem Detecting financial market manipulation activities like spoofing and layering.
method Modeling order book dynamics as particle motion and using momentum measure.
result Method outperforms conventional Z-score-based anomaly detection.
Theory explains how deep nets learn features from data.
problem Understanding how deep neural networks learn features from data.
method Developed a noise-nonlinearity phase diagram and a mechanical theory.
result Links feature learning across layers to generalization.
We study soft persistence (existence in subsequent temporal layers of motifs from the initial layer) of motif structures in Triangulated Maximally Filtered Graphs (TMFG) generated from time-varying Kendall correlation matrices computed from stock prices log-returns over rolling windows with exponential smoothing. We ob…
FLoE adapts LLMs by selectively deploying LoRA adapters based on layer importance and task requirements.
problem Uniform LoRA deployment across all layers leads to inefficient and redundant parameter allocation.
method FLoE uses Fisher information to dynamically identify task-critical layers and optimizes LoRA ranks.
result FLoE achieves significant efficiency-accuracy trade-offs, especially in resource-constrained environments.
We analyse a multiplex of networks between OECD countries during the decade 2002-2010, which consists of five financial layers, given by foreign direct investment, equity securities, short-term, long-term and total debt securities, and five environmental layers, given by emissions of N O x, P M 10 SO 2, CO 2 equivalent…
We explore and expand the Soft Nearest Neighbor Loss to measure the entanglement of class manifolds in representation space: i.e., how close pairs of points from the same class are relative to pairs of points from different classes. We demonstrate several use cases of the loss. As an analytical to…
In the analysis of machine learning models, it is often convenient to assume that the parameters are IID. This assumption is not satisfied when the parameters are updated through training processes such as SGD. A relaxation of the IID condition is a probabilistic symmetry known as exchangeability. We show the sense in …
Paper analyzes infinite-width attention layers using Tensor Programs.
problem Capturing the infinite-width limit of attention layers.
method Tensor Programs framework to rigorously identify the limit distribution.
result Derives exact form of infinite-width limit distribution without Gaussian approximations.
Structure inference is an important task for network data processing and analysis in data science. In recent years, quite a few approaches have been developed to learn the graph structure underlying a set of observations captured in a data space. Although real-world data is often acquired in settings where relationship…
Study on neural networks' sample complexity with one hidden layer.
problem Understanding how sample complexity is affected by network architecture and norm constraints.
method Norm-based uniform convergence bounds for scalar-valued one-hidden-layer networks, focusing on spectral and Frobenius norms.
result Spectral norm control is insufficient for uniform convergence guarantees, but Frobenius norm control is sufficient, with conditions.