Objects are represented in sensory systems by continuous manifolds due to sensitivity of neuronal responses to changes in physical features such as location, orientation, and intensity. What makes certain sensory representations better suited for invariant decoding of objects by downstream networks? We present a theory…
Echo state network (ESN) is viewed as a temporal non-orthogonal expansion with pseudo-random parameters. Such expansions naturally give rise to regressors of various relevance to a teacher output. We illustrate that often only a certain amount of the generated echo-regressors effectively explain the variance of the tea…
New explanation of reservoir computing using random projections.
problem Understanding the randomness in reservoir computing.
method Constructing strongly universal reservoir systems as random projections of state-space systems.
result Approximation of any fading memory filters class by training a linear readout for each filter.
New measures link neural representation geometry to decoding ability.
problem Understanding how neural representations relate to decoding ability.
method Showed that popular similarity measures can be interpreted from a decoding perspective.
result Proved that measures like CKA and CCA quantify alignment between optimal linear readouts.
A new readout layer improves graph property prediction.
problem Graph property prediction challenges due to information loss.
method Introduces a novel generalized global pooling layer for Message-Passing Neural Networks.
result New state-of-the-art results in graph property prediction.
GNNs improve semi-supervised node regression, but why? We explain.
problem Understanding when and why GNNs succeed in semi-supervised node regression.
method Aggregate-and-readout model encompassing message passing architectures, least-squares estimation over GNNs with linear graph convolutions and a deep ReLU readout.
result Sharp non-asymptotic risk bound separating approximation, stochastic, and optimization errors.
This work establishes universality for deep equivariant networks, overcoming limitations of previous approaches.
problem Rarity of universality results for equivariant neural networks, especially in high-dimensional settings.
method Develops a more general account of universality for equivariant networks, introducing entry-wise separability and readout layers.
result Deep equivariant networks achieve universality under entry-wise separability, with or without readout layers.
New reservoir computing approach handles infinite-dimensional systems.
problem Approximating and generalizing complex input/output systems.
method Randomly generated echo state networks with neural networks.
result Proves universal approximation properties for new class of systems.
We propose reinforcement learning on simple networks consisting of random connections of spiking neurons (both recurrent and feed-forward) that can learn complex tasks with very little trainable parameters. Such sparse and randomly interconnected recurrent spiking networks exhibit highly non-linear dynamics that transf…
Quantum reservoirs risk bounds are analyzed using Rademacher complexity.
problem Bounding generalization errors of quantum reservoirs.
method Using Rademacher complexity, specific bounds are derived for quantum reservoir classes.
result Risk bounds converge with increasing training samples and qubits.
We propose an approach to learning with graph-structured data in the problem domain of graph classification. In particular, we present a novel type of readout operation to aggregate node features into a graph-level representation. To this end, we leverage persistent homology computed via a real-valued, learnable, filte…
Random feature models approximate functions in Banach spaces efficiently.
problem Approximating functions in Banach spaces efficiently.
method Randomly initialized feature maps and linear readout training.
result Universal approximation in Bochner spaces for Banach space-valued models.
We present a novel neural network algorithm, the Tensor Switching (TS) network, which generalizes the Rectified Linear Unit (ReLU) nonlinearity to tensor-valued hidden units. The TS network copies its entire input vector to different locations in an expanded representation, with the location determined by its hidden un…
New insights into continual learning with task similarity.
problem Challenges in learning similar tasks without interference.
method Linear teacher-student model with latent structure.
result High input feature similarity with low readout similarity is catastrophic.
Universal approximation theorem for differentiable maps on infinite-dimensional manifolds
problem Approximation of differentiable maps on infinite-dimensional manifolds
method Weighted universal approximation theorem
result Universal approximation theorem for differentiable maps
Deep networks improve by progressively refining approximations at each layer.
problem Standard approximation theory doesn't explain the role of intermediate layers in deep neural networks.
method Developed a mixed-activation architecture with a geometric scale interpretation of depth.
result Each intermediate layer approximates the target function with a geometric rate.
This work optimizes reservoir computing models by linking recurrence and non-linear dynamics.
problem Understanding how recurrence and non-linear dynamics in cortical networks contribute to their function.
method Transformed time-continuous, recurrent dynamics into an effective feed-forward structure of linear and non-linear temporal kernels.
result Optimal time-series classifiers can be built from random reservoir networks, demonstrating significant performance gains.
Generalizes neural network approximation to infinite-dimensional manifolds and derivatives.
problem Approximating differentiable maps on infinite-dimensional manifolds.
method Proves a weighted Nachbin theorem to establish universal approximation for differentiable maps, including derivatives.
result Linear functions of the signature can approximate path space functionals including their derivatives.
MV-GNN improves molecular property prediction by integrating atom and bond information.
problem Accurately predicting molecular properties using graph neural networks.
method Multi-View Graph Neural Network (MV-GNN) architecture with shared self-attentive readout and cross-dependent message passing.
result MV-GNN achieves superior performance on molecular property prediction benchmarks.
Measures mode separation in high-dimensional densities via a reversible diffusion process.
problem Quantifying how sharply a distribution fragments into barrier-separated clusters in high dimensions.
method A unique reversible diffusion process with f as stationary distribution, extracting SSA and DA from its autocovariance matrix.
result Empirical autocovariance spectrum and readouts (SSA, DA) quantify mode separation using only samples and pretrained score-based models.
New framework assesses neural sensitivity to small perturbations.
problem Comparing neural representations' sensitivity to small changes.
method Local decodable information, Fisher information, and projected pullback/Fisher metric.
result Reveals differences in neural sensitivity not captured by activation alignment.
Study shows depth improves generalization in deep learning models.
problem Understanding why and when depth improves generalization in deep learning.
method Implementation-agnostic state-transition model to analyze depth and generalization.
result Identifies geometric and semigroup mechanisms that keep entropy contribution saturated or polynomial, clarifying depth's statistical advantage.
Reservoir Computing (RC) refers to a Recurrent Neural Networks (RNNs) framework, frequently used for sequence learning and time series prediction. The RC system consists of a random fixed-weight RNN (the input-hidden reservoir layer) and a classifier (the hidden-output readout layer). Here we focus on the sequence lear…
RFMs transition from linear to nonlinear under specific input-label correlation.
problem Understanding the transition from linear to nonlinear behavior in RFMs.
method Analyzing RFMs under spiked covariance designs, characterizing the interaction between anisotropy and input-label correlation.
result The RFM generalization error is governed by the strength of input-label correlation, leading to a clear nonlinear advantage above a specific boundary.
We study how finite Bayesian neural networks adapt their hidden representations.
problem Understanding how finite Bayesian neural networks differ from infinite ones.
method We analyze the asymptotics of learned feature kernels for various network architectures.
result The leading finite-width corrections to feature kernels have a universal form.
Training deep neural networks with the error backpropagation algorithm is considered implausible from a biological perspective. Numerous recent publications suggest elaborate models for biologically plausible variants of deep learning, typically defining success as reaching around 98% test accuracy on the MNIST data se…
The paper explores how AI trading agents' similar information representation can cause financial market instability.
problem Systemic instability in AI-dominated financial markets due to similar information representation.
method Structural multi-agent market model with two-layer decision architecture for AI agents.
result Representation homogeneity can lead to systemic instability in financial markets.
New method combines long-memory reservoirs for accurate dengue forecasting from short data.
problem Accurate dengue forecasting from short, noisy, non-stationary, and nonlinear data.
method Fractional ESN and Wavelet ESN frameworks integrating long-term memory.
result fESN and wESN outperform baselines in multiple dengue datasets and forecasting horizons.
Echo State Networks (ESN) are a class of Recurrent Neural Networks (RNN) that has gained substantial popularity due to their effectiveness, ease of use and potential for compact hardware implementation. An ESN contains the three network layers input, reservoir and readout where the reservoir is the truly recurrent netw…
Functional input neural networks approximate continuous functions on weighted spaces.
problem Approximating continuous functions on infinite-dimensional weighted spaces.
method Additive family mapping, non-linear activation, linear readouts, Stone-Weierstrass theorem.
result Global universal approximation of continuous functions on weighted spaces.
A new framework for efficient sequence maps using Bayesian filtering and covariance.
problem Designing efficient recurrent sequence maps from explicit memory assumptions.
method Design-model framework, exact Bayesian filtering, query-dependent readout, linear-Gaussian instantiation.
result Improved robustness and retrieval performance across various benchmarks.
The study quantifies how many objects can be linearly classified under all views.
problem Understanding the expressivity of group-equivariant representations.
method Generalization of Cover's Function Counting Theorem to quantify separable dichotomies.
result The fraction of separable dichotomies is determined by the fixed space dimension of the group action.
LoCo learns local representations without end-to-end synchronization, improving performance on complex tasks.
problem Learning local representations without end-to-end synchronization constraints.
method Overlap local blocks to increase decoder depth and allow feedback from upper to lower layers.
result LoCo closes the performance gap between local learning and end-to-end contrastive learning.
Characterizes test error in learning with deep, structured feature maps.
problem Characterizing test error in learning with deep, structured feature maps.
method Asymptotic analysis of feature covariance and population covariance.
result Closed-form formula for feature covariance in Gaussian rainbow neural networks.
The mushroom body is the key network for the representation of learned olfactory stimuli in Drosophila and insects. The sparse activity of Kenyon cells, the principal neurons in the mushroom body, plays a key role in the learned classification of different odours. In the specific case of the fruit fly, the sparseness o…
Procedure removes training data dependency from deep networks, improving generalization.
problem Removing dependency on training data in deep networks for better generalization.
method Deterministic and stochastic parts to ensure forgetting, leveraging activation and weight dynamics.
result New bound on information extraction from black-box networks, ensuring forgetting in activations.
We derive scaling laws for optimizing neural networks in hardware.
problem Optimizing the large parameter space of neural networks in hardware.
method Analytical derivation of scaling laws for Coordinate Descent optimization.
result Convergence is exponential and scales linearly with the number of neurons.
We analyze generalization in deep learning models using random matrix theory.
problem Understanding the generalization error in deep learning models with random feature representations.
method Applying Random Matrix Theory to derive asymptotic generalization error formulas for various architectures.
result Linear ESNs are equivalent to ridge regression with exponentially time-weighted input covariance, revealing an inductive bias towards recent inputs.
One of the goals of the ICML workshop on representation and learning is to establish benchmark scores for a new data set of labeled facial expressions. This paper presents the performance of a "Null" model consisting of convolutions with random weights, PCA, pooling, normalization, and a linear readout. Our approach fo…
We extend manifold capacity to nonlinear neural representations with contextual information.
problem Efficient processing of information through neural representations.
method Theoretical framework leveraging latent directions in input space related to contextual information.
result Derivation of an exact formula for context-dependent manifold capacity.
A new compressed sensing system speeds up PSTE detection.
problem Accurately detecting fast, small optical events (PSTEs) is challenging.
method Developed a novel compressed sensing algorithm for rolling shutter systems.
result Accurately recovered PSTEs with spatial undersampling, showing speed and quality advantages.
Softmax is a standard final layer used in Neural Nets (NNs) to summarize information encoded in the trained NN and return a prediction. However, Softmax leverages only a subset of the class-specific structure encoded in the trained model and ignores potentially valuable information: During training, models encode an ar…
The reservoir computing paradigm is employed to classify heartbeat anomalies online based on electrocardiogram signals. Inspired by the principles of information processing in the brain, reservoir computing provides a framework to design, train, and analyze recurrent neural networks (RNNs) for processing time-dependent…
Semantic paraphrases can fool financial sentiment classifiers due to geometric shifts in model representations.
problem Semantic paraphrase attacks on financial sentiment classifiers
method Developing a continuous local model of semantic paraphrase perturbations
result The worst-case local displacement of the target representation is governed by the largest generalised eigenvalue of a matrix pencil constructed from the Jacobians of the two embedding maps.
A framework evaluates the impact of different modules in graph contrastive learning.
problem Insufficient module-level evaluation in existing graph contrastive learning methods.
method Proposes a framework decomposing GCL models into four modules for module-level evaluation.
result Identifies module-level guidelines and competitive performance of different modules.
New conditions ensure deep neural networks can approximate any function on non-Euclidean spaces.
problem Understanding how to modify neural network architectures to approximate functions on non-Euclidean spaces.
method Developed conditions for feature and readout maps that preserve universal approximation capabilities.
result Modified architectures can deterministically approximate any classifier on non-Euclidean spaces.
The study analyzes transfer learning in infinite-width neural networks, improving generalization on target tasks.
problem Improving generalization in neural networks when using pretraining on a source task.
method Developed a theory under gradient flow for infinitely wide networks, analyzing fine-tuning and joint pretraining.
result Summary statistics of randomly initialized networks after pretraining are adaptive kernels that depend on both source and target data.
Simplified image clustering achieves competitive results without text-based embeddings.
problem Complexity and resource requirements of state-of-the-art clustering methods.
method SCP: trains a small cluster head using pre-trained vision model features and positive data pairs.
result SCP achieves highly competitive performance on various benchmark datasets.