Modular neural networks generalize better with less data.
problem Theoretical and practical understanding of how modularity improves neural network generalization.
method Theoretical analysis of sample complexity, development of a novel learning rule.
result Modular networks require fewer samples to generalize compared to nonmodular networks, especially in high-dimensional tasks.
Neural networks learn modular arithmetic but not all, extending known solutions to generalize.
problem Neural networks struggle with modular arithmetic, especially for polynomials.
method Developed analytical solutions for MLP networks to learn modular addition and multiplication, then combined these solutions to generalize on arbitrary modular polynomials.
result Neural networks can learn and generalize solutions to modular polynomials, supporting the hypothesis that some polynomials are learnable.
Artificial neural networks (ANNs) have achieved significant success in tackling classical and modern machine learning problems. As learning problems grow in scale and complexity, and expand into multi-disciplinary territory, a more modular approach for scaling ANNs will be needed. Modular neural networks (MNNs) are neu…
The study examines if ReLU activation function is optimal for modularity in neural networks.
problem Finding the best activation function for modularity in neural networks.
method Comparing ReLU with other activation functions for modularity and performance.
result ReLU may not be the best choice for modularity, suggesting other functions could be more suitable.
Noise-driven neural networks emerge modular structures, improving robustness and generalization.
problem Artificial neural networks struggle with modular solutions, leading to poor generalization and robustness.
method Inspired by brain's modular architecture, the study uses neural noise and nonlinear responses to drive the emergence of modular solutions.
result Noise-driven modularisation improves robustness and generalization in neural networks.
New methods detect modular structure in neural networks, revealing surprising effects of dropout.
problem Detecting functional modules in neural networks for learning, compositionality, and generalization.
method Two families of methods: upstream and downstream, to define similarity between units.
result Dropout dramatically increased modularity, and there's little agreement between upstream and downstream methods.
Training a Neural Network (NN) with lots of parameters or intricate architectures creates undesired phenomena that complicate the optimization process. To address this issue we propose a first modular approach to NN design, wherein the NN is decomposed into a control module and several functional modules, implementing …
Study explores how neural networks and Transformers learn modular arithmetic with multiple inputs.
problem Understanding how neural networks and Transformers learn modular arithmetic with multiple inputs.
method Analytical characterization of features learned by neural networks and Transformers, focusing on margin maximization and Fourier spectra.
result Neural networks and Transformers require a minimum neuron count of \( m \geq 2^{2k-2} \cdot (p-1) \) to solve modular addition problems with \( k \) inputs and modulus \( p \).
Recursive Feature Machines show grokking in modular arithmetic without neural networks.
problem Grokking in modular arithmetic tasks.
method Recursive Feature Machines (RFM) with Average Gradient Outer Product (AGOP).
result RFM and neural networks learn block-circulant features to solve modular arithmetic.
RNNs solve modular addition tasks using low rank and sparse Fourier structures.
problem Solving modular addition tasks with recurrent neural networks.
method Identified low rank structures and sparse Fourier representations in RNN weights.
result RNNs robust to removing individual frequencies but degrade with more ablation.
Sine activation functions enable two-layer neural networks to learn modular addition more efficiently.
problem Learning modular addition with two-layer neural networks.
method Introduced and analyzed sine activation functions, providing theoretical and empirical evidence.
result Sine activation functions allow for constant-width network realizations of modular addition, whereas ReLU networks require linear width scaling.
A new neural network for efficient density estimation.
problem Efficient density estimation for high-dimensional data.
method Triangular neural network implementation of neural autoregressive flow (NAF).
result Achieves state-of-the-art bits-per-dimension indices on MNIST and CIFAR-10.
Unified policy controls diverse agents through modular neural networks.
problem Learning control policies for various agent morphologies.
method Shared Modular Policies (SMP) with decentralized control and message passing.
result A single modular policy controls multiple agent morphologies.
MNN improves American call option pricing accuracy.
problem Inaccurate valuation of American call options.
method Modular Neural Network (MNN) model.
result MNN model outperforms traditional models and FNN.
New theory maps neural network weights to optimize faster and scale.
problem Optimizing neural networks for speed and scalability.
method Constructing a duality map using layer-wise operator norms.
result Derived GPU-friendly algorithms for various layers.
PICLE uses probabilistic models to efficiently evaluate and compose modules for continual learning.
problem Challenging search space of module compositions in continual learning.
method Probabilistic framework to cheaply compute module compositions' fitness.
result First modular CL algorithm to achieve perceptual, few-shot, and latent transfer.
Modular neural causal models outperform other models in generalization and adaptation.
problem Robust out-of-distribution generalization and fast adaptation in machine learning.
method Factorizing data generating process into modules using only causal parents as predictors.
result Modular neural causal models offer robust generalization and fast adaptation, especially in low data regimes.
We explain how neural networks learn to solve modular addition tasks.
problem How two-layer neural networks learn to solve modular addition tasks.
method Formalized a diversification condition during training, proving it allows the network to approximate the correct logic for modular addition.
result Neural networks can robustly identify the correct sum through phase symmetry and frequency diversification.
An artificial agent for financial risk and returns' prediction is built with a modular cognitive system comprised of interconnected recurrent neural networks, such that the agent learns to predict the financial returns, and learns to predict the squared deviation around these predicted returns. These two expectations a…
Modular meta-learning is a new framework that generalizes to unseen datasets by combining a small set of neural modules in different ways. In this work we propose abstract graph networks: using graphs as abstractions of a system's subparts without a fixed assignment of nodes to system subparts, for which we would need …
Neuro-inspired recurrent neural network algorithms, such as echo state networks, are computationally lightweight and thereby map well onto untethered devices. The baseline echo state network algorithms are shown to be efficient in solving small-scale spatio-temporal problems. However, they underperform for complex task…
Layered neural networks have greatly improved the performance of various applications including image processing, speech recognition, natural language processing, and bioinformatics. However, it is still difficult to discover or interpret knowledge from the inference provided by a layered neural network, since its inte…
BNNpriors library improves Bayesian neural network inference with various prior distributions.
problem Challenges in choosing good prior distributions for Bayesian neural networks.
method State-of-the-art Markov Chain Monte Carlo inference with a wide range of predefined priors.
result Facilitates foundational discoveries on the nature of the cold posterior effect.
Objective The electrical characteristics of the EEG signals can be used for seizure detection. Statistical independence between different brain regions is measured by functional brain connectivity (FBC). Specific directional effects can't consider by FBC and thus effective brain connectivity (EBC) is used to measure ca…
Lifelong learning is a very important step toward realizing robust autonomous artificial agents. Neural networks are the main engine of deep learning, which is the current state-of-the-art technique in formulating adaptive artificial intelligent systems. However, neural networks suffer from catastrophic forgetting when…
Scaling model capacity has been vital in the success of deep learning. For a typical network, necessary compute resources and training time grow dramatically with model size. Conditional computation is a promising way to increase the number of parameters with a relatively small increase in resources. We propose a train…
Improved recurrent neural networks learn long-term dependencies through multi-scale memory.
problem Capturing long-term dependencies in recurrent neural networks.
method Incremental training of a modular RNN architecture with multi-scale hidden states.
result Incremental training and multi-scale memory enhance RNNs' ability to learn long-term dependencies.
ARMA cell simplifies neural autoregressive modeling for time series.
problem Complex RNN cells are not always necessary and can be inferior.
method Introduces ARMA cell, a simpler, modular approach for neural time series modeling.
result The ARMA cell is competitive with popular alternatives in performance.
Graph clustering remains challenging for GNNs, but a new method improves performance.
problem Graph clustering is difficult for GNNs, especially in noisy data.
method Developed Deep Modularity Networks (DMoN) inspired by modularity measure.
result DMoN produces high-quality clusters with over 40% improvement over other methods.
In this paper we take a problem of unsupervised nodes clustering on graphs and show how recent advances in attention models can be applied successfully in a "hard" regime of the problem. We propose an unsupervised algorithm that encodes Bethe Hessian embeddings by optimizing soft modularity loss and argue that our mode…
Attention mechanism combines bottom-up and top-down signals in neural networks.
problem Combining robust perception with bottom-up and top-down signals.
method Attention mechanism over modulated recurrent neural networks.
result Bidirectional information flow leads to improved performance in various tasks.
Interpreting the prediction mechanism of complex models is currently one of the most important tasks in the machine learning field, especially with layered neural networks, which have achieved high predictive performance with various practical data sets. To reveal the global structure of a trained neural network in an …
This paper compares hypernetworks and embedding methods for function approximation.
problem Learning to map inputs to functions efficiently for each instance.
method Compared embedding-based and hypernetwork methods using expressivity and complexity.
result Hypernetworks can be more efficient in terms of trainable parameters for certain functions.
New method calibrates complex ODEs from noisy data using neural networks.
problem Calibrating multi-dimensional complex ODEs from noisy data.
method Two-stage approach: de-noising and higher-order derivatives followed by a deep neural network.
result Consistent recovery of ODE system without curse of dimensionality.
POLICE enforces linear constraints on deep neural networks efficiently.
problem Enforcing constraints on deep neural networks without affecting optimization.
method Provably optimal affine constraint enforcement method that minimally modifies DNNs.
result POLICE ensures DNNs fulfill affine constraints during training and testing.
PyTorch Frame simplifies multi-modal tabular learning with modular data and model handling.
problem Handling complex multi-modal tabular data in deep learning.
method A PyTorch-based framework that provides a data structure, model abstraction, and integration with external models.
result Demonstrated the effectiveness of PyTorch Frame in implementing and applying diverse tabular models to complex multi-modal tabular data.
Proposes a CL technique to improve accuracy and reduce forgetting.
problem Sequential task learners struggle with forgetting information from previous tasks.
method Extracts modular parts of neural networks and estimates task relatedness.
result Remarkable performance gain in robustness to forgetting for EWC and GEM methods.
Modular deep learning framework using pairwise labels without backpropagation.
problem Efficiently training deep neural networks with limited supervision.
method Stacked linear models in feature spaces, provably optimal modular learning framework.
result High accuracy (94.88%) achieved with minimal labeled examples (1 per class).
To deepen our understanding of graph neural networks, we investigate the representation power of Graph Convolutional Networks (GCN) through the looking glass of graph moments, a key property of graph topology encoding path of various lengths. We find that GCNs are rather restrictive in learning graph moments. Without c…
NOMU improves neural network uncertainty estimation.
problem Estimating model uncertainty for neural networks with limited data.
method Introduces NOMU, a two-sub-NN architecture with a designed loss function.
result NOMU outperforms state-of-the-art methods in regression and Bayesian optimization.
Flexible framework for deep distributional regression models.
problem Learning conditional distributions from semi-structured data.
method Combines additive regression models with deep networks using TensorFlow.
result State-of-the-art predictive performance with interpretability.
Improved graph clustering with modularity and coarsening for attributes and communities.
problem Inaccurate community detection and computational inefficiency in graph clustering.
method Integrates coarsening and modularity maximization, using a loss function with log-determinant, smoothness, and modularity components.
result Superior clustering outcomes, proven consistent under DC-SBM, and efficient algorithm integration with GNNs and VGAEs.
Persistent homology reveals a topological signature of grokking in neural networks.
problem Understanding how neural networks learn and generalize from modular arithmetic tasks.
method Persistent homology on point clouds derived from embedding matrices of models trained on modular arithmetic.
result A sharp increase in first homology persistence indicates grokking, with a dominant long-lived topological feature and structured secondary features.
New GCNs solve graph embedding problems efficiently and interpretably.
problem Graph embedding for scalable and interpretable machine learning.
method Proposed two GCNs: CAFE-GCN and sphere-GCN, based on constrained optimization.
result Both GCNs yield good approximations of dominant eigenvectors and perform dimensionality reduction.
A Graph Neural Network model for generating molecular graphs.
problem Designing new drug molecules efficiently and cost-effectively.
method Sequential molecular graph generator based on Graph Neural Networks.
result The model can generate molecular graphs without overfitting and outperforms existing methods.
We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian). The approach reduces the otherwise tedious manual derivation of these mat…
Many prediction problems, such as those that arise in the context of robotics, have a simplifying underlying structure that, if known, could accelerate learning. In this paper, we present a strategy for learning a set of neural network modules that can be combined in different ways. We train different modular structure…
Modularity is a popular measure of community structure. However, maximizing the modularity can lead to many competing partitions, with almost the same modularity, that are poorly correlated with each other. It can also produce illusory "communities" in random graphs where none exist. We address this problem by using th…