Superposition accelerates training to a universal power-law exponent.
problem Training dynamics in neural networks.
method Teacher-student framework and analytic theory.
result Superposition leads to a universal power-law exponent of ~1, independent of data and channel statistics.
Paper develops a simple estimator for high-dimensional superposition models with various component structures.
problem Estimating high-dimensional superposition models with different component structures.
method Presented a simple estimator for general superposition models with any number of component parameters and any norm structure.
result Geometric condition and high probability non-asymptotic bounds for accurate component estimation.
Improved learning of Hawkes processes with superposition-assisted optimization.
problem Learning multi-agent Hawkes processes with shared and different intensities.
method Stochastic optimization with superposition-driven diversity strategy.
result Superposition improves risk bound and convergence properties.
Study harmonic surfaces in 3D space, proving superposition principle.
problem Understanding harmonic surfaces in R3. method Using harmonic Enneper immersions and superposition principle.
result Minimal and maximal surfaces can be decomposed into harmonic components.
In many learning tasks, structural models usually lead to better interpretability and higher generalization performance. In recent years, however, the simple structural models such as lasso are frequently proved to be insufficient. Accordingly, there has been a lot of work on "superposition-structured" models where mul…
Paper introduces Manifold Probe for discovering representation manifolds in superposition.
problem Discovering representation manifolds in complex superposition representations.
method Generalizes linear regression probes to learn feature spaces and directions in superposition representations.
result Demonstrates Manifold Probe on Llama 2-7b representations, finding causally involved manifolds in model behaviour.
New methods combine model predictions to avoid linear mixtures' limitations.
problem Combining predictions from different models to avoid linear mixtures' limitations.
method Log-linear pooling (locking) and quantum superposition (quacking) to optimise model weights.
result Demonstrated locking method with illustrative example and practical application.
New algorithm extracts features from superpositions in machine learning models.
problem Challenges in extracting interpretable features from complex models in superposition.
method An efficient query algorithm that identifies non-degenerate feature directions and reconstructs the function.
result Identifies all feature directions whose responses are non-degenerate and reconstructs the function \( f \) in a general superposition setting.
Superposition in autoencoders leads to loss in simple models.
problem Mechanistic interpretability of neural networks
method Analyzing mathematical basis for superposition and providing bounds for reconstruction loss
result Upper and lower bounds for L2 reconstruction loss in the sparse regime
Superposition rules form a class of functions that describe general solutions of systems of first-order ordinary differential equations in terms of generic families of particular solutions and certain constants. In this work we extend this notion and other related ones to systems of higher-order differential equations …
Paper proposes an algorithm to reconstruct optimal model structure from graph adjacency matrix.
problem Optimal model structure reconstruction from weighted colored graph adjacency matrix.
method Uses prize-collecting Steiner tree algorithm to reconstruct minimum spanning tree.
result Demonstrates the effectiveness of the prize-collecting Steiner tree algorithm for model structure reconstruction.
Mixed superposition rules, i.e., functions describing the general solution of a system of first-order differential equations in terms of a generic family of particular solutions of first-order systems and some constants, are studied. The main achievement is a generalization of the celebrated Lie-Scheffers Theorem, char…
The paper develops methods to derive mixed superposition rules for Lie systems and applies them to various physical systems.
problem Finding general solutions for Lie systems.
method Develops mixed superposition rules for Lie systems with imprimitive Lie algebras and semidirect sums.
result Extends coalgebra method to Lie systems of partial differential equations.
Quantum machine learning uses superposition to create a large ensemble of classifiers.
problem Improving machine learning efficiency on quantum computers.
method Using superposition to create an exponentially large ensemble of classifiers, trained with an optimization-free learning algorithm.
result Adding an optimization step improves the performance of quantum ensembles of classifiers.
Formulae for Bäcklund transformations of hyperbolic and elliptic sine-Gordon/sinh-Gordon equations.
problem Finding solutions for specific types of equations.
method Providing superposition formulae for Bäcklund transformations.
result Algebraically obtain infinitely many solutions after first integration.
We consider the class of integer rectifiable currents without boundary satisfying a positivity condition. We establish that these currents can be written as a linear superposition of graphs of finitely many functions with bounded variation.
Observing a linear superposition principle, a family of new minimal hypersurfaces in Euclidean space is found, as well as that linear combinations of generalized helicoids induce new algebraic minimal cones of arbitrarily high degree.
Quantum computing speeds up asset pricing models exponentially.
problem Solving dynamic nonlinear asset pricing models efficiently.
method Utilizes quantum superposition and entanglement to solve models exponentially faster than classical methods.
result Exponential computational speed-up for solving asset pricing models.
MIMONets speed up neural network inference by processing multiple inputs in parallel.
problem Reducing computational cost in neural network inference for large datasets.
method Proposes MIMONets, which augment neural network architectures with variable binding mechanisms to handle multiple inputs in superposition.
result Achieves significant speedups (2-4x) with minimal accuracy loss, demonstrating adaptability across different architectures.
The study examines Euclid's Book I, focusing on area applications and construction methods.
problem Exploring Euclid's geometric constructions and proofs, particularly those involving area calculations.
method Summarizing medieval editions and ancient commentaries, comparing constructions and proofs.
result Medieval editions often avoid Euclid's use of superposition in area proofs, offering alternative constructions.
SupSup model learns thousands of tasks without forgetting, using randomly initialized subnetworks.
problem Sequentially learning many tasks without forgetting.
method Randomly initialized base network with task-specific subnetworks (supermasks).
result Gradient-based optimization can identify the correct subnetwork for new tasks.
Sparse superposition codes were recently introduced by Barron and Joseph for reliable communication over the AWGN channel at rates approaching the channel capacity. The codebook is defined in terms of a Gaussian design matrix, and codewords are sparse linear combinations of columns of the matrix. In this paper, we prop…
A theory of feature geometry using spectral analysis of weight matrices.
problem Current methods decompose neural network activations into sparse linear features, losing geometric structure.
method Develops a theory by analyzing the spectra of weight-derived matrices, introducing the frame operator.
result Features collapse onto single eigenspaces, organizing into tight frames, and admit discrete classification.
Gravitational instantons are constructed as superpositions of Atiyah-Hitchin and Taub-NUT geometries.
problem Constructing gravitational instantons from Atiyah-Hitchin and Taub-NUT geometries.
method A gluing construction that captures the superposition of moduli spaces of centred SU(2) monopoles and Taub-NUT manifolds.
result Gravitational instantons are explicitly shown to be superpositions of Atiyah-Hitchin and Taub-NUT geometries.
We analyze families of non-autonomous systems of first-order ordinary differential equations admitting a common time-dependent superposition rule, i.e., a time-dependent map expressing any solution of each of these systems in terms of a generic set of particular solutions of the system and some constants. We next study…
Resonator Networks solve high-dimensional vector factorization better than optimization methods.
problem High-dimensional vector factorization problem in Vector Symbolic Architectures.
method Recurrent neural network (Resonator Networks) that combines nonlinear dynamics and superposition search.
result Resonator Networks outperform optimization methods in solving high-dimensional vector factorization.
Extends quasi-Lie systems to PDEs for integrability analysis.
problem Analyzing integrability conditions for PDEs.
method Develops a procedure to construct quasi-Lie systems for PDEs through quasi-Lie schemes.
result Obtains t-dependent superposition rules and integrability conditions. Method recovers structured components from nonlinear observations.
problem Demixing structured vectors from nonlinear observations.
method Proposes a method to recover components from nearly m = O(s) samples.
result Strictly improves upon previous techniques and matches best sample complexity.
Superposed Hawkes processes improve risk bounds and solve cold-start issues.
problem Improving risk bounds in temporal point processes.
method Least squares estimation of superposed Hawkes processes.
result Superposed Hawkes processes tighten risk bounds under certain conditions.
Explores tensor products in hyperdimensional computing.
problem Understanding tensor products in hyperdimensional computing.
method Generalized results from graph embeddings to vector symbolic architectures and hyperdimensional computing.
result Tensor product is the most general and expressive representation with errorless unbinding and detection.
Curves in 3-sphere with constant curvature and torsion are described.
problem Understanding curves with constant curvature and torsion in a 3-sphere.
method Analyzing the motion of a point with superposition of circular motions.
result Behavior of these curves can be periodic or dense in a Clifford torus.
Algorithm recovers components from few samples of high-dimensional vectors with structured sparsity.
problem Demixing high-dimensional vectors from few samples with structured sparsity.
method Iterative thresholding algorithm for stable component recovery.
result Algorithm provably recovers components with n=O(s) samples, achieving fast convergence and per-iteration complexity. Lie systems method simplifies Riccati hierarchy study.
problem Simplifying study of Riccati hierarchy equations.
method Lie systems approach to projective Riccati equations.
result Characterization of Riccati chain equations geometrically.
Quantum algorithms for multi-armed bandits are explored with limited reward access.
problem Exploring quantum speed-ups in multi-armed bandit problems with limited reward information.
method Introduced new bandit models and showed query complexity equivalence with classical algorithms.
result No quadratic speed-up is possible for multi-armed bandits with limited reward access.
This paper proves a version for stochastic differential equations of the Lie-Scheffers Theorem. This result characterizes the existence of nonlinear superposition rules for the general solution of those equations in terms of the involution properties of the distribution generated by the vector fields that define it. Wh…
Study new k-contact distributions and Lie systems.
problem Characterizing and understanding k-contact distributions. method Analyzing Goursat distributions and Lie systems.
result Characterized new types of k-contact distributions. Develops methods to answer counterfactual questions in temporal point processes.
problem Lack of counterfactual analysis in temporal point process models.
method Causal model of thinning based on Gumbel-Max structural causal model, superposition theorem, and sampling algorithm.
result Simulation of counterfactual realizations provides valuable insights for targeted interventions.
Paper proposes GAN frameworks for learning clean signals from superposed structured components.
problem Learning clean signals from superposed structured components when clean samples are not available.
method Proposes denoising-GAN and demixing-GAN frameworks to learn the structure of components.
result Demonstrates competitive performance in tasks like denoising, demixing, and compressive sensing.
New GAN model deblends galaxy images with high accuracy and speed.
problem Deblending blended galaxy images in dense regions of the universe.
method Branched generative adversarial network (GAN) to produce images of deblended galaxies.
result High peak signal-to-noise ratio and structural similarity scores compared to ground truth images.
We show that Scherk's first surface, a one-parameter family of solutions to the minimal surface equation, may be written as a linear superposition of other solutions with specific parametric values.
Simpler proof for non-basic sets in 2D.
problem Proving non-basic sets in 2D.
method Defining Sternfeld arrays and proving non-basic sets.
result Simpler proof of non-basic sets in 2D.
ProSper learns data components with non-standard priors and superpositions.
problem Learning complex data components with non-standard priors and superpositions.
method Probabilistic algorithms for sparse coding with non-standard priors and superpositions.
result Library supports scalable and parallelizable dictionary learning for large-scale applications.
Defines manifolds of mappings between function spaces and discusses their properties.
problem Defining smooth manifolds of mappings between function spaces.
method Defines a smooth manifold structure on sets of continuous mappings and discusses properties of natural mappings.
result Properties of spaces of sections and smoothness of natural mappings between spaces of mappings.
We fit the volatility fluctuations of the S&P 500 index well by a Chi distribution, and the distribution of log-returns by a corresponding superposition of Gaussian distributions. The Fourier transform of this is, remarkably, of the Tsallis type. An option pricing formula is derived from the same superposition of Black…
Automated method finds meaningful directions in neural network activations.
problem Mixed selectivity in neurons makes interpretation challenging.
method Automated quantification of interpretability and discovery of meaningful directions.
result Meaningful directions in neural network activations are more interpretable than individual neurons.
We present a probabilistic model of events in continuous time in which each event triggers a Poisson process of successor events. The ensemble of observed events is thereby modeled as a superposition of Poisson processes. Efficient inference is feasible under this model with an EM algorithm. Moreover, the EM algorithm …
Improves early stopping in deep networks by adjusting stepsizes.
problem Epoch-wise double descent in deep networks.
method Analytical and empirical study of bias-variance tradeoffs in different network layers.
result Eliminating epoch-wise double descent through adjusting stepsizes of different layers improves early stopping performance.
It is shown that superpositions of path integrals with arbitrary Hamiltonians and different scaling parameters v ("variances") obey the Chapman-Kolmogorov relation for Markovian processes if and only if the corresponding smearing distributions for v have a specific functional form. Ensuing "smearing" distributions subs…