The paper improves neural network robustness against multiple norm types of adversarial attacks.
problem Defending neural networks against adversarial attacks with different norms.
method Combining existing defense mechanisms to train neural networks robust against both ℓ∞ and ℓ2 attacks. result New defense mechanisms offer better protection against both ℓ∞ and ℓ2 attacks. New mechanisms from primate vision improve neural network robustness.
problem Demonstrating robust neural networks to small adversarial perturbations.
method Investigated two biologically plausible mechanisms: non-uniform retina sampling and receptive field diversity.
result Non-uniform retina sampling and receptive field diversity improve adversarial robustness.
A new mechanism for deep learning privacy preserving.
problem Preserving differential privacy in deep learning models.
method Adaptive Laplace mechanism for deep neural networks.
result The mechanism optimally balances privacy and model performance.
Deep neural networks can store and recall data efficiently.
problem Identifying computational mechanisms for memorization and retrieval of data.
method Training overparameterized autoencoders and sequence encoders using standard optimization methods.
result Overparameterized autoencoders and sequence encoders store and recall data efficiently as attractors.
Paper identifies how neural networks learn features.
problem Understanding how neural networks automatically select features.
method Deep Neural Feature Ansatz, implementing average gradient outer product.
result Recursive Feature Machines achieve state-of-the-art performance.
Statistical mechanics reveals phase transitions in ε-SVR error.
problem Understanding task precision in neural representations with variability.
method Statistical mechanics applied to ε-SVR. result Double-descent phenomenon in generalization error due to ε. Paper introduces a Gaussian Process for operator learning in computational mechanics.
problem Efficient and accurate solutions for large datasets with reliable uncertainty quantification.
method Gaussian Process (GP) embedded in a neural operator framework with stochastic dual descent (SDD) algorithm.
result Improves GP resolution independence and scalability for high-dimensional and non-linear systems.
Permutation-equivariant neural networks improve auction mechanisms by reducing regret and sample complexity.
problem Designing optimal auction mechanisms that balance revenue and bidders' regret.
method Introduced permutation-equivariant neural networks to auction mechanisms.
result Permutation-equivariant neural networks decrease expected ex-post regret and improve model generalizability.
Paper introduces methods to integrate external knowledge into RNNs using attention mechanisms.
problem Incorporating external knowledge into RNNs for improved performance.
method Proposes three methods: attentional concatenation, feature-based gating, and affine transformation.
result Attentional feature-based gating consistently improves performance across tasks.
Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input. We propose in this paper a new framework for sparse and structured attention, building upon a smoothed max operator. We show that the gradient of this operator defines a mapping from real val…
New mechanism protects neural network weights from privacy attacks during self-supervised learning.
problem Privacy risks during fine-tuning stage of self-supervised learning.
method Proposes a novel differential privacy mechanism using additive logistic noise.
result Reduces membership inference attack accuracy to 50% while maintaining below 5% performance loss.
New method uses neural networks to solve statistical mechanics problems.
problem Statistical mechanics of systems with finite size.
method Variational autoregressive neural networks with reinforcement learning.
result Directly computes free energy, entropy, magnetizations, and correlations.
New neural network improves audio classification accuracy.
problem Challenging audio classification problem in pattern recognition.
method Introduces a Classifier-Attention-Based Convolutional Neural Network (CAB-CNN) with an attention mechanism to reduce classifier complexity.
result Significantly improves audio classification performance, achieving more than 10% improvements.
New approach uses statistical mechanics to explain deep learning generalization.
problem Understanding deep learning's generalization properties.
method Revisiting statistical mechanics in neural networks, introducing control parameters.
result Simple model explains overfitting, discontinuous learning, and sharp transitions.
A neural network-evolutionary framework estimates mechanical RUL efficiently.
problem Estimating the remaining useful life of mechanical systems.
method Multi-layer perceptron and evolutionary algorithm for optimizing data parameters, using strided time windows.
result The framework increases model efficiency and reduces complexity, leading to improved accuracy.
Paper proposes a method to interpret deep neural networks using attention mechanisms.
problem Interpreting deep neural network models to understand their performance.
method Proposes a novel method using attention mechanisms to analyze neural network models.
result Improved attention based method shows better classifier interpretation.
This paper makes complex neural graphs nearly convex through iterative decomposition and scale mechanism.
problem Non-convexity in complex neural architectures limits their performance in convex optimization.
method Decompose neural graph into operators, algorithms, and functions; iteratively propagate along edges; introduce scale mechanism to transform non-convex properties.
result Proves neural graph is nearly convex in each variable when others are fixed, validating the scale mechanism.
The paper introduces a pooling mechanism for graph CNNs using NMF.
problem Pooling in graph structured data for efficient computation.
method Non-negative matrix factorization for node pooling.
result The pooling mechanism improves graph classification performance.
Analyzes unsupervised neural networks using statistical mechanics and Monte Carlo simulations.
problem Understanding computational capabilities of unsupervised neural networks.
method Statistical mechanics approach and Monte Carlo simulations.
result Obtained a phase diagram summarizing network performance.
New statistical mechanics analysis shows edge pruning outperforms node pruning in neural networks.
problem Theoretical understanding of neural network pruning effectiveness is lacking.
method Statistical mechanics analysis of a teacher-student framework.
result DPP node pruning method is superior to other methods, but edge pruning is better overall.
Deep neural network improves NILM with attention mechanism.
problem Energy disaggregation of individual appliance power demands from aggregate meter readings.
method Regression and classification subnetworks with attention mechanism.
result Proposed model outperforms state-of-the-art on REDD and UK-DALE datasets.
DNNs generalize object recognition in novel orientations via neurons tuned to common features.
problem Understanding how DNNs generalize to objects in novel orientations.
method Training DNNs with familiar objects from multiple viewpoints and analyzing neuron responses.
result DNNs disseminate orientation-invariance from familiar objects to recognize objects in novel orientations.
A hybrid neural network improves robustness in estimating vehicle parameters from noisy data.
problem Estimating parameters of a mechanical vehicle model from noisy acceleration data.
method Introduced a convolutional neural network with two objective functions: naive and hybrid.
result The hybrid objective function outperforms the naive one in robustness on noisy input data.
Physics-informed neural networks improve model accuracy and efficiency.
problem Accurate dynamic models for technical systems are hard to achieve.
method Physics-informed neural ordinary differential equations (PINODE) integrating Lagrangian mechanics.
result Hybrid model combines physical insight and data approximation.
Replicated attention model in image classification and fine-grained recognition.
problem Improving attention mechanisms in neural networks.
method Implemented the 'Learn to Pay Attention' model in convolutional neural networks.
result Successfully replicated results in image classification and fine-grained recognition.
A new neural network learns optimal designs in one shot.
problem Designing optimal structures in iterative processes.
method Theory-driven machine learning combining student and teacher models.
result Near-optimal designs achieved with less data.
Noether's framework reveals symmetry-breaking in neural networks.
problem Understanding the role of symmetry breaking in neural networks.
method Developed a theoretical framework using Lagrangian mechanics.
result Identified 'kinetic symmetry breaking' and its effect on learning dynamics.
Neural network speeds up atmospheric chemistry modeling 4250x.
problem Computational expense of simulating atmospheric chemistry.
method Created a neural network to emulate a complex chemical mechanism.
result Achieved a 250x computational speedup.
New attention mechanism improves meta-transfer learning in dynamic tasks.
problem Underfitting in meta-transfer learning with dynamic tasks.
method Proposed Recurrent Memory Reconstruction (RMR) attention mechanism.
result ASNP-RMR significantly outperforms baselines in various tasks.
This study approximates neural network features for modeling relations and attention mechanisms.
problem Approximating neural network features for modeling relations and attention mechanisms.
method Analyzes inner products of multi-layer perceptrons for universal approximation of symmetric and asymmetric relation functions.
result Universal approximation of relation functions and attention mechanisms using inner products of neural networks.
The study connects deep neural networks with statistical mechanics, revealing natural activation functions.
problem Understanding the activation functions in deep neural networks.
method Statistical Mechanics model of deep neural networks, focusing on encoding, validation, and propagation steps.
result A set of natural activations including Sigmoid, tanh, ReLU, and Swish are identified.
Attention-based CNNs improve band selection in hyperspectral images.
problem Selecting informative bands from hyperspectral images for accurate classification.
method Attention-based convolutional neural networks reusing activations at different depths.
result Deep models with attention mechanisms achieve high-quality classification and identify significant bands.
Deep models learn biases from datasets, hindering understanding of binding mechanisms.
problem Dataset biases prevent deep models from revealing fragment logic of protein-ligand binding.
method Attribution method to identify and exploit dataset biases in neural networks.
result Deep models can be fooled into learning spurious correlations from biased datasets.
A new multi-layer attention mechanism improves speech keyword recognition accuracy.
problem Inaccurate attention weights in LSTM networks for speech keyword recognition.
method Introducing information from layers prior to feature extraction into attention weights calculations.
result The proposed multi-layer attention mechanism leads to more accurate attention weights and improved keyword spotting performance.
A caching mechanism improves sequence to logical form generation accuracy.
problem Generating logical forms from natural language sequences.
method Proposes a caching mechanism to increase output probability of source input tokens and weigh them based on context.
result Improves sequence/token-level accuracy on sequence to logical form tasks.
Physics insights into deep learning mechanisms.
problem Understanding the internal workings of deep learning models.
method Examining deep learning from quantum mechanics, statistical physics, and physical world perspectives.
result Provides new theoretical insights and practical applications for deep learning.
Develops a complexity measure for neural networks based on quantum statistical mechanics.
problem Understanding the relationship between neural network structure and generalization ability.
method Introduces Periodic Spectral Ergodicity (PSE) and cascading PSE (cPSE) to quantify neural network complexity.
result Demonstrates the effectiveness of cPSE in quantifying complexity and guiding NAS.
New mechanism discovered for feature learning in CNNs.
problem Understanding how CNNs learn features from images.
method Proposed Convolutional Neural Feature Ansatz linking filter covariances to patch-based AGOPs.
result Deep ConvRFM algorithm learns features similar to deep CNNs, improving performance.
New model prevents forgetting in continual learning.
problem Learning from a continuous stream of tasks without forgetting.
method Bayesian Optimized Continual Learning with Attention Mechanism (BOCL).
result BOCL outperforms state-of-the-art in preventing catastrophic forgetting and fitting new tasks better.
Enhances statistical mechanics solving using VANs with MCMC or importance sampling.
problem Sampling error in solving statistical mechanics using VANs.
method Integrates MCMC or importance sampling to correct sampling error in VANs.
result Asymptotically unbiased estimators for physical quantities are achieved.
AttViz offers online visualizations of neural language model attention mechanisms.
problem Limited interpretability of neural language models.
method Online toolkit for exploring self-attention mechanisms.
result Visualizations help understand model decision-making.
Efficient trainable front-end for neural speech enhancement.
problem Inefficient STFT front-ends in neural speech enhancement models.
method Butterfly mechanism for Fast Fourier Transform, trainable STFT window.
result Accuracy and efficiency improvements for low-compute systems.
APD method decomposes neural network parameters into simple, faithful components.
problem Understanding the internal mechanisms learned by neural networks.
method Attribution-based Parameter Decomposition (APD) method.
result Demonstrated effectiveness in recovering features, separating computations, and identifying representations.
CPA models improve GNNs by preserving node cardinality.
problem Limited understanding of attention-based GNNs' discriminative power.
method Theoretical analysis and CPA models to preserve cardinality information.
result CPA models can improve GNNs' performance in node and graph classification.
We extend Hopfield neural networks using mechanical analogy for better handling of interactions.
problem Improving the stability of spurious states in Hopfield neural networks.
method Relativistic extension via mechanical analogy, focusing on low-storage regime.
result Reduced stability of spurious states due to unlearning contributions.
Proposes Neural SDE for better model robustness and generalization.
problem Missing regularization mechanisms in Neural ODE networks.
method Integrates various regularization mechanisms via stochastic noise injection.
result Improves robustness and generalization compared to Neural ODE.
Convolutional neural network improves MRE image reconstruction.
problem Reconstructing MRE images from displacement data is computationally intensive and costly.
method Proposes a CNN architecture to directly map MRE displacement data into elastograms, introducing a secondary loss for training.
result CNN-generated images compare favorably with nonlinear inversion methods.
EML-CD discovers causal mechanisms from neural networks in a structured way.
problem Extracting causal mechanisms from neural network weights is ill-posed.
method Integrates EML operator into causal structure learning, representing each edge mechanism as a gated EML binary tree.
result Achieves SHD=11.2 +/- 0.4 on real data, matching or outperforming existing methods.