Neural networks learn modular arithmetic but not all, extending known solutions to generalize.
problem Neural networks struggle with modular arithmetic, especially for polynomials.
method Developed analytical solutions for MLP networks to learn modular addition and multiplication, then combined these solutions to generalize on arbitrary modular polynomials.
result Neural networks can learn and generalize solutions to modular polynomials, supporting the hypothesis that some polynomials are learnable.
Survey of methods to train deep architectures without E2EBP.
problem Training deep architectures without end-to-end backpropagation.
method Modular training, weakly modular hybrids.
result Provably optimal alternatives to E2EBP can match or surpass E2EBP performance.
The article is devoted to the qR-conformal modular functors, which being ``deformations'' of the conformal modular functor (the projective representation of the category Train(Diff+(S1)), the train of the group Diff+(S1) of all orientation preserving diffeomorphisms of a circle) in the class of all projectiv…
Training a Neural Network (NN) with lots of parameters or intricate architectures creates undesired phenomena that complicate the optimization process. To address this issue we propose a first modular approach to NN design, wherein the NN is decomposed into a control module and several functional modules, implementing …
Modular neural networks generalize better with less data.
problem Theoretical and practical understanding of how modularity improves neural network generalization.
method Theoretical analysis of sample complexity, development of a novel learning rule.
result Modular networks require fewer samples to generalize compared to nonmodular networks, especially in high-dimensional tasks.
A framework for modular training of robust generative models.
problem Training large generative models is resource-intensive and requires heuristic tuning.
method Modular training using a gating mechanism and a minimax game to find a robust gate.
result The modular approach can theoretically outperform monolithic baselines and is scalable.
RNNs solve modular addition tasks using low rank and sparse Fourier structures.
problem Solving modular addition tasks with recurrent neural networks.
method Identified low rank structures and sparse Fourier representations in RNN weights.
result RNNs robust to removing individual frequencies but degrade with more ablation.
Modular deep learning framework using pairwise labels without backpropagation.
problem Efficiently training deep neural networks with limited supervision.
method Stacked linear models in feature spaces, provably optimal modular learning framework.
result High accuracy (94.88%) achieved with minimal labeled examples (1 per class).
Noise-driven neural networks emerge modular structures, improving robustness and generalization.
problem Artificial neural networks struggle with modular solutions, leading to poor generalization and robustness.
method Inspired by brain's modular architecture, the study uses neural noise and nonlinear responses to drive the emergence of modular solutions.
result Noise-driven modularisation improves robustness and generalization in neural networks.
Unified policy controls diverse agents through modular neural networks.
problem Learning control policies for various agent morphologies.
method Shared Modular Policies (SMP) with decentralized control and message passing.
result A single modular policy controls multiple agent morphologies.
Proposes efficient deep causal generative models for high-dimensional causal inference.
problem Inefficient training of deep generative models for high-dimensional data.
method Modular training of deep causal generative models using adversarial training and pre-trained models.
result First algorithm that provably samples from any identifiable causal query in the presence of latent confounders.
Recursive Feature Machines show grokking in modular arithmetic without neural networks.
problem Grokking in modular arithmetic tasks.
method Recursive Feature Machines (RFM) with Average Gradient Outer Product (AGOP).
result RFM and neural networks learn block-circulant features to solve modular arithmetic.
Study shows how transformers learn to combine simple tasks into complex ones.
problem Understanding how transformers learn to perform complex tasks not seen during training.
method Controlled setting involving variable assignment and modular addition; partitioned training data analysis.
result Small transformers can generalize to unseen combinations of variables and numbers.
Transformers learn to solve modular arithmetic tasks by in-context learning and skill composition.
problem Understanding how large language models generalize to unseen tasks in modular arithmetic.
method Pre-training on a set of modular arithmetic tasks and evaluating out-of-distribution performance.
result Transformers require two transformer blocks for out-of-distribution generalization, and deeper models exhibit transient out-of-distribution performance.
New methods for scalable inference in modular models with misspecified sub-models.
problem Model misspecification in multi-modular models complicates evidence combination.
method Variational methods for approximating Cut and SMI posteriors, and Variational Meta-Posterior.
result Feasibility of analysis with multiple cuts using a single set of variational parameters.
Ordering examples in modular arithmetic training affects model performance.
problem Training efficiency and model performance in modular arithmetic.
method Varying example ordering while keeping other conditions constant.
result Adversarially structured ordering suppresses learning, while fixed-ordering strategies achieve high accuracy.
The study examines if ReLU activation function is optimal for modularity in neural networks.
problem Finding the best activation function for modularity in neural networks.
method Comparing ReLU with other activation functions for modularity and performance.
result ReLU may not be the best choice for modularity, suggesting other functions could be more suitable.
Theoretical analysis explains why models generalize after overfitting in modular addition.
problem Understanding why models generalize after overfitting in modular addition.
method Theoretical analysis and gradient descent behavior of two-layer quadratic networks and Transformers.
result Two-layer quadratic networks and simple Transformers generalize well after initially overfitting, indicating grokking.
PICLE uses probabilistic models to efficiently evaluate and compose modules for continual learning.
problem Challenging search space of module compositions in continual learning.
method Probabilistic framework to cheaply compute module compositions' fitness.
result First modular CL algorithm to achieve perceptual, few-shot, and latent transfer.
Soft modularization improves sample efficiency and performance in reinforcement learning.
problem Challenges in training multiple tasks jointly in reinforcement learning.
method Explicit modularization technique on policy representation, soft modularization method.
result Improves sample efficiency and performance over strong baselines in robotics manipulation tasks.
Modular meta-learning is a new framework that generalizes to unseen datasets by combining a small set of neural modules in different ways. In this work we propose abstract graph networks: using graphs as abstractions of a system's subparts without a fixed assignment of nodes to system subparts, for which we would need …
Modular RL modules solve complex 3D Sokoban tasks.
problem Solving complex, integrated tasks combining visual, physical, and abstract reasoning.
method Compose RL modules in a sense-plan-act hierarchy, using only model-free methods.
result Modular RL outperforms state-of-the-art monolithic RL on Mujoban.
New theory maps neural network weights to optimize faster and scale.
problem Optimizing neural networks for speed and scalability.
method Constructing a duality map using layer-wise operator norms.
result Derived GPU-friendly algorithms for various layers.
Modularity-aware GAE and VGAE improve community detection and link prediction.
problem Improving community detection with GAE and VGAE in the absence of node features.
method Introducing a modularity-aware message passing scheme and regularizer to GAE and VGAE encoders.
result Jointly addressing community detection and link prediction with high accuracy is possible.
Scaling model capacity has been vital in the success of deep learning. For a typical network, necessary compute resources and training time grow dramatically with model size. Conditional computation is a promising way to increase the number of parameters with a relatively small increase in resources. We propose a train…
In this paper we take a problem of unsupervised nodes clustering on graphs and show how recent advances in attention models can be applied successfully in a "hard" regime of the problem. We propose an unsupervised algorithm that encodes Bethe Hessian embeddings by optimizing soft modularity loss and argue that our mode…
A modular GP framework for efficient transfer learning.
problem Efficiently transfer knowledge across different tasks or datasets.
method Modular variational Gaussian processes (GPs) with a dictionary of well-fitted GPs.
result Reduces computational costs and allows the transfer of uncertainty metrics.
NACs learn modular neural architectures without domain knowledge.
problem Jointly learn module configuration and execution without domain knowledge.
method Jointly trains two systems: module configuration and execution.
result Improves low-shot adaptation and OOD robustness.
We explain how neural networks learn to solve modular addition tasks.
problem How two-layer neural networks learn to solve modular addition tasks.
method Formalized a diversification condition during training, proving it allows the network to approximate the correct logic for modular addition.
result Neural networks can robustly identify the correct sum through phase symmetry and frequency diversification.
We introduce SLM Lab, a software framework for reproducible reinforcement learning (RL) research. SLM Lab implements a number of popular RL algorithms, provides synchronous and asynchronous parallel experiment execution, hyperparameter search, and result analysis. RL algorithms in SLM Lab are implemented in a modular w…
We find and propose an explanation for a large variety of modularity-related symmetries in problems of 3-manifold topology and physics of 3d N=2 theories where such structures a priori are not manifest. These modular structures include: mock modular forms, SL(2,Z) Weil representations, quantum mo…
Researchers found the global topology of the Eisenstein-Picard modular surface.
problem Understanding the global topology of the Eisenstein-Picard modular surface.
method Quotient space of the complex hyperbolic plane by the modular group.
result Determined the global topology of the Eisenstein-Picard modular surface as a 4-orbifold.
Study modular surfaces in Lorentz-Minkowski 3-space, classifying and analyzing their curvature and applications.
problem Understanding the curvature properties of modular surfaces in Lorentz-Minkowski space.
method Analyzing the sign of Gaussian and mean curvature, classifying surfaces, and applying to conformal field theories.
result Complete classification of zero Gaussian curvature modular surfaces and non-existence of non-planar maximal modular surfaces.
Study modular forms over Γ^0(2) and anomaly cancellation formulas.
problem Anomaly cancellation formulas for modular forms over Γ^0(2).
method Study and analysis of modular forms over Γ^0(2).
result Anomaly cancellation formulas derived for modular forms over Γ^0(2).
Our aim is to introduce and advocate non-Σ (non-symmetric) modular operads. While ordinary modular operads were inspired by the structure of the moduli space of stable complex curves, non-Σ modular operads model surfaces with open strings outputs. An immediate application of our theory is a short proof that the mod…
We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian). The approach reduces the otherwise tedious manual derivation of these mat…
Many prediction problems, such as those that arise in the context of robotics, have a simplifying underlying structure that, if known, could accelerate learning. In this paper, we present a strategy for learning a set of neural network modules that can be combined in different ways. We train different modular structure…
An artificial agent for financial risk and returns' prediction is built with a modular cognitive system comprised of interconnected recurrent neural networks, such that the agent learns to predict the financial returns, and learns to predict the squared deviation around these predicted returns. These two expectations a…
Fuchsian groups with a modular embedding have the richest arithmetic properties among non-arithmetic Fuchsian groups. But they are very rare, all known examples being related either to triangle groups or to Teichmueller curves. In Part I of this paper we study the arithmetic properties of the modular embedding and deve…
Improved algorithm for modular links provides upper volume bounds.
problem Understanding the geometry of modular links and Lorenz links.
method Bunch algorithm to study modular links and provide upper volume bounds.
result First upper volume bound independent of word exponents and quadratic in braid index.
Geodesics on modular surface yield arithmetic 3-manifolds.
problem Understanding arithmetic properties of modular surfaces.
method Constructing geodesics and analyzing their lifts.
result Complements of canonical lifts are arithmetic 3-manifolds.
We introduce the notion of the modular class of a Lie algebroid equipped with a Nambu structure. In particular, we recover the modular class of a Nambu-Poisson manifold M with its Nambu tensor Λ as the modular class of the tangent Lie algebroid TM with Nambu structure Λ. We show that many known properties of th…
New modular forms for anomaly cancellation formulas on any dimensional manifolds.
problem Constructing new modular forms for anomaly cancellation formulas.
method Using E8 bundles, constructing modular forms on any dimensional manifolds. result Derived new anomaly cancellation formulas and applications.
Holomorphic networks on modular arithmetic show clear success or failure, no in-between.
problem Understanding when neural networks can represent modular arithmetic tasks.
method Two-layer networks with holomorphic monomial activations trained on modular tasks.
result The network's output is confined to a subspace of characters, and representability depends on the task's Fourier support.
Quantum modularity proved for SU(2) TQFT signature on genus 2 surfaces.
problem Proving quantum modularity of SU(2) TQFT signature for genus 2 surfaces.
method Using quantum modularity of generalized Dedekind sums associated with modular forms and trigonometric sum expressions.
result Quantum modularity of SU(2) TQFT signature on genus 2 surfaces proved.
MOCA uses modular attention to estimate causal effects from complex data.
problem Estimating causal effects from observational data with complex, non-linear, and high-dimensional treatment and outcome mechanisms.
method MOCA is a transformer-based framework that separates treatment and outcome modeling through modular design and one-way attention mechanism, with cutting-feedback to prevent outcome influence on treatment representations.
result MOCA outperforms classical estimators and machine learning approaches across various simulated and real-world scenarios.
Motivated by a question of Hirzebruch on the possible topological types of cusp cross-sections of Hilbert modular varieties, we give a necessary and sufficient condition for a manifold M to be diffeomorphic to a cusp cross-section of a Hilbert modular variety. Specialized to Hilbert modular surfaces, this proves that e…
Countable modular groups found on surfaces with infinite type.
problem Finding modular groups of infinite type surfaces.
method Proving countable modular groups for orientable infinite type surfaces.
result Every orientable infinite type surface has a countable modular group.