Modular neural networks generalize better with less data.
problem Theoretical and practical understanding of how modularity improves neural network generalization.
method Theoretical analysis of sample complexity, development of a novel learning rule.
result Modular networks require fewer samples to generalize compared to nonmodular networks, especially in high-dimensional tasks.
Neural networks learn modular arithmetic but not all, extending known solutions to generalize.
problem Neural networks struggle with modular arithmetic, especially for polynomials.
method Developed analytical solutions for MLP networks to learn modular addition and multiplication, then combined these solutions to generalize on arbitrary modular polynomials.
result Neural networks can learn and generalize solutions to modular polynomials, supporting the hypothesis that some polynomials are learnable.
A new modularity density measure improves community detection in heterogeneous networks.
problem Detecting meaningful communities in heterogeneous networks.
method Formulated a novel metric, modularity density, for undirected, weighted networks.
result Maximization of modularity density is free from bias and better at detecting weakly-separated communities.
Artificial neural networks (ANNs) have achieved significant success in tackling classical and modern machine learning problems. As learning problems grow in scale and complexity, and expand into multi-disciplinary territory, a more modular approach for scaling ANNs will be needed. Modular neural networks (MNNs) are neu…
Noise-driven neural networks emerge modular structures, improving robustness and generalization.
problem Artificial neural networks struggle with modular solutions, leading to poor generalization and robustness.
method Inspired by brain's modular architecture, the study uses neural noise and nonlinear responses to drive the emergence of modular solutions.
result Noise-driven modularisation improves robustness and generalization in neural networks.
The study examines if ReLU activation function is optimal for modularity in neural networks.
problem Finding the best activation function for modularity in neural networks.
method Comparing ReLU with other activation functions for modularity and performance.
result ReLU may not be the best choice for modularity, suggesting other functions could be more suitable.
New methods detect modular structure in neural networks, revealing surprising effects of dropout.
problem Detecting functional modules in neural networks for learning, compositionality, and generalization.
method Two families of methods: upstream and downstream, to define similarity between units.
result Dropout dramatically increased modularity, and there's little agreement between upstream and downstream methods.
Training a Neural Network (NN) with lots of parameters or intricate architectures creates undesired phenomena that complicate the optimization process. To address this issue we propose a first modular approach to NN design, wherein the NN is decomposed into a control module and several functional modules, implementing …
RNNs solve modular addition tasks using low rank and sparse Fourier structures.
problem Solving modular addition tasks with recurrent neural networks.
method Identified low rank structures and sparse Fourier representations in RNN weights.
result RNNs robust to removing individual frequencies but degrade with more ablation.
Modularity is a popular measure of community structure. However, maximizing the modularity can lead to many competing partitions, with almost the same modularity, that are poorly correlated with each other. It can also produce illusory "communities" in random graphs where none exist. We address this problem by using th…
Soft modularization improves sample efficiency and performance in reinforcement learning.
problem Challenges in training multiple tasks jointly in reinforcement learning.
method Explicit modularization technique on policy representation, soft modularization method.
result Improves sample efficiency and performance over strong baselines in robotics manipulation tasks.
Sine activation functions enable two-layer neural networks to learn modular addition more efficiently.
problem Learning modular addition with two-layer neural networks.
method Introduced and analyzed sine activation functions, providing theoretical and empirical evidence.
result Sine activation functions allow for constant-width network realizations of modular addition, whereas ReLU networks require linear width scaling.
New method uses hyperspherical geometry to improve community detection.
problem Improving community detection methods in network analysis.
method Mapping networks to points on a hypersphere, then projecting to clustering vectors.
result Modularity maximization is equivalent to minimizing angular distance on the hypersphere.
Theoretical analysis explains why models generalize after overfitting in modular addition.
problem Understanding why models generalize after overfitting in modular addition.
method Theoretical analysis and gradient descent behavior of two-layer quadratic networks and Transformers.
result Two-layer quadratic networks and simple Transformers generalize well after initially overfitting, indicating grokking.
Study explores how neural networks and Transformers learn modular arithmetic with multiple inputs.
problem Understanding how neural networks and Transformers learn modular arithmetic with multiple inputs.
method Analytical characterization of features learned by neural networks and Transformers, focusing on margin maximization and Fourier spectra.
result Neural networks and Transformers require a minimum neuron count of \( m \geq 2^{2k-2} \cdot (p-1) \) to solve modular addition problems with \( k \) inputs and modulus \( p \).
Unified policy controls diverse agents through modular neural networks.
problem Learning control policies for various agent morphologies.
method Shared Modular Policies (SMP) with decentralized control and message passing.
result A single modular policy controls multiple agent morphologies.
Revealing a community structure in a network or dataset is a central problem arising in many scientific areas. The modularity function Q is an established measure quantifying the quality of a community, being identified as a set of nodes having high modularity. In our terminology, a set of nodes with positive modular…
New modularity function improves clustering of spatially embedded networks.
problem Improving clustering in spatially embedded networks for unsupervised learning.
method Developed a new modularity function and compared its performance with existing methods.
result Our modularity function outperforms existing methods in partitioning 2D and 3D granular assemblies.
Recursive Feature Machines show grokking in modular arithmetic without neural networks.
problem Grokking in modular arithmetic tasks.
method Recursive Feature Machines (RFM) with Average Gradient Outer Product (AGOP).
result RFM and neural networks learn block-circulant features to solve modular arithmetic.
Dynamic information balancing reduces catastrophic forgetting in modular neural networks.
problem Catastrophic forgetting in neural networks when learning multiple tasks.
method Dynamic Information Balancing (DIB) using reinforcement learning to adaptively route inputs based on module information load.
result DIB combined with EWC regularization outperforms models with similar capacity and EWC regularization.
PICLE uses probabilistic models to efficiently evaluate and compose modules for continual learning.
problem Challenging search space of module compositions in continual learning.
method Probabilistic framework to cheaply compute module compositions' fitness.
result First modular CL algorithm to achieve perceptual, few-shot, and latent transfer.
A new model for detecting overlapping communities in weighted networks.
problem Community detection in overlapping weighted networks with mixed membership and edge weights.
method Mixed membership distribution-free (MMDF) model with an efficient spectral algorithm and fuzzy weighted modularity.
result The MMDF model can estimate community memberships and evaluate community quality for weighted networks.
GCNs struggle with learning graph moments, but modular designs improve their performance.
problem GCNs' limitations in learning graph moments.
method Investigated through graph moments, analyzed expressiveness, designed modular GCNs.
result Modular GCNs using different propagation rules can distinguish graphs from various models.
Modular meta-learning is a new framework that generalizes to unseen datasets by combining a small set of neural modules in different ways. In this work we propose abstract graph networks: using graphs as abstractions of a system's subparts without a fixed assignment of nodes to system subparts, for which we would need …
In this paper we analyse the bipartite Colombian firms-products network, throughout a period of five years, from 2010 to 2014. Our analysis depicts a strongly modular system, with several groups of firms specializing in the export of specific categories of products. These clusters have been detected by running the bipa…
Neuro-inspired recurrent neural network algorithms, such as echo state networks, are computationally lightweight and thereby map well onto untethered devices. The baseline echo state network algorithms are shown to be efficient in solving small-scale spatio-temporal problems. However, they underperform for complex task…
Networks capture pairwise interactions between entities and are frequently used in applications such as social networks, food networks, and protein interaction networks, to name a few. Communities, cohesive groups of nodes, often form in these applications, and identifying them gives insight into the overall organizati…
We present a novel clustering approach for moving object trajectories that are constrained by an underlying road network. The approach builds a similarity graph based on these trajectories then uses modularity-optimization hiearchical graph clustering to regroup trajectories with similar profiles. Our experimental stud…
A new neural network for efficient density estimation.
problem Efficient density estimation for high-dimensional data.
method Triangular neural network implementation of neural autoregressive flow (NAF).
result Achieves state-of-the-art bits-per-dimension indices on MNIST and CIFAR-10.
Study the Mexican stock market's interdependency structure from 2000-2019.
problem Characterize the interdependency structure of the Mexican Stock Exchange.
method Estimate correlation/concentration matrices from different models and compute network theory metrics.
result Visualizations provide a comprehensive overview of the stock market's interdependency structure.
We explain how neural networks learn to solve modular addition tasks.
problem How two-layer neural networks learn to solve modular addition tasks.
method Formalized a diversification condition during training, proving it allows the network to approximate the correct logic for modular addition.
result Neural networks can robustly identify the correct sum through phase symmetry and frequency diversification.
MNN improves American call option pricing accuracy.
problem Inaccurate valuation of American call options.
method Modular Neural Network (MNN) model.
result MNN model outperforms traditional models and FNN.
Modular deep learning framework using pairwise labels without backpropagation.
problem Efficiently training deep neural networks with limited supervision.
method Stacked linear models in feature spaces, provably optimal modular learning framework.
result High accuracy (94.88%) achieved with minimal labeled examples (1 per class).
New theory maps neural network weights to optimize faster and scale.
problem Optimizing neural networks for speed and scalability.
method Constructing a duality map using layer-wise operator norms.
result Derived GPU-friendly algorithms for various layers.
Paper proposes a new unsupervised clustering method using attention models.
problem Unsupervised community detection on graphs.
method Optimizes soft modularity loss on Bethe Hessian embeddings.
result Model performs competitively with classical and GNN methods.
An artificial agent for financial risk and returns' prediction is built with a modular cognitive system comprised of interconnected recurrent neural networks, such that the agent learns to predict the financial returns, and learns to predict the squared deviation around these predicted returns. These two expectations a…
Networks are a convenient way to represent complex systems of interacting entities. Many networks contain "communities" of nodes that are more densely connected to each other than to nodes in the rest of the network. In this paper, we investigate the detection of communities in temporal networks represented as multilay…
ProMoD models human race drivers with probabilistic movement primitives and neural networks.
problem Challenging task of modeling human driver behavior due to variability and complexity.
method Modular framework with Probabilistic Movement Primitives, clothoids, and neural networks.
result Significant advantages in imitation accuracy and robustness compared to other algorithms.
SDQL uses modular deep Q networks to efficiently learn multi-stage optimal control tasks.
problem Training complex deep reinforcement learning models for multi-stage control tasks is inefficient and unstable.
method Stacked Deep Q Learning (SDQL) with modular Q networks and backward training.
result SDQL efficiently learns optimal control policies for multi-stage tasks with high-dimensional state and action spaces.
PyTorch Frame simplifies multi-modal tabular learning with modular data and model handling.
problem Handling complex multi-modal tabular data in deep learning.
method A PyTorch-based framework that provides a data structure, model abstraction, and integration with external models.
result Demonstrated the effectiveness of PyTorch Frame in implementing and applying diverse tabular models to complex multi-modal tabular data.
Proposes a CL technique to improve accuracy and reduce forgetting.
problem Sequential task learners struggle with forgetting information from previous tasks.
method Extracts modular parts of neural networks and estimates task relatedness.
result Remarkable performance gain in robustness to forgetting for EWC and GEM methods.
Improved graph clustering with modularity and coarsening for attributes and communities.
problem Inaccurate community detection and computational inefficiency in graph clustering.
method Integrates coarsening and modularity maximization, using a loss function with log-determinant, smoothness, and modularity components.
result Superior clustering outcomes, proven consistent under DC-SBM, and efficient algorithm integration with GNNs and VGAEs.
Modular neural causal models outperform other models in generalization and adaptation.
problem Robust out-of-distribution generalization and fast adaptation in machine learning.
method Factorizing data generating process into modules using only causal parents as predictors.
result Modular neural causal models offer robust generalization and fast adaptation, especially in low data regimes.
Clustering on hypergraphs has been garnering increased attention with potential applications in network analysis, VLSI design and computer vision, among others. In this work, we generalize the framework of modularity maximization for clustering on hypergraphs. To this end, we introduce a hypergraph null model, analogou…
Develops deep neural networks for accurate seizure detection.
problem Seizure detection using EEG signals and brain connectivity.
method Combines EBC, DMNN, MENN, and MFNN methods.
result DMNN achieves highest accuracy (99.43%).
The study shows how modular learning can adapt to new tasks.
problem Adapting to new tasks in an ever-changing environment.
method Task segmentation, modular learning, memory-based ensembling.
result The system demonstrates robustness to catastrophic forgetting and increasing positive transfer.
Recursive sketches summarize deep networks, aiding quick analysis and learning.
problem Understanding and analyzing complex deep learning models.
method Developed a recursive sketch mechanism to summarize inputs and outputs of modular deep networks.
result Sketches can identify key components and summarize essential information, even if partially erased.
Torch-Points3D simplifies 3D deep learning research and reproducibility.
problem Lack of transparency and reproducibility in 3D deep learning research.
method Modular framework with quality-of-life features, standardized protocols, and open-source implementation.
result Facilitates fair and rigorous evaluation of 3D deep learning methods.