The study examines how averaging data improves model performance.
problem Understanding the generalization gap in machine learning models.
method Data averaging, covariance analysis, and stochastic gradient descent (SGD) noise modeling.
result A modified generalization gap is always non-negative for a large class of model parameter distributions.
Gated attention improves model curvature, enhancing performance on nonlinear tasks.
problem Understanding the geometric implications of gating in attention mechanisms.
method Modeling attention outputs as Gaussian distributions and analyzing Fisher--Rao geometry.
result Gated attention enables non-flat geometries, including positively curved manifolds.
New volume functions for random hyperbolic surfaces link to spectral gaps.
problem Analyzing spectral gaps in random hyperbolic surfaces.
method Introduced new volume functions VgT(l), derived their asymptotic expansions, and linked them to spectral gaps. result Coefficients in the asymptotic expansion of VgT(l) are Friedman-Ramanujan functions. New framework explains normalizing flows' power and limitations.
problem Understanding the expressive power and limitations of normalizing flows.
method Theoretical framework for well-conditioned coupling-based normalizing flows and volume-preserving flows.
result RealNVP is distributionally universal, but volume-preserving flows are not.
The motivations for using variational inference (VI) in neural networks differ significantly from those in latent variable models. This has a counter-intuitive consequence; more expressive variational approximations can provide significantly worse predictions as compared to those with less expressive families. In this …
New algorithm reduces sample complexity for planning in MDPs.
problem Planning in MDPs with unknown transitions.
method MDP-GapE, a trajectory-based MCTS algorithm.
result Proves upper bound on sample complexity in terms of sub-optimality gaps.
Sine activation functions enable two-layer neural networks to learn modular addition more efficiently.
problem Learning modular addition with two-layer neural networks.
method Introduced and analyzed sine activation functions, providing theoretical and empirical evidence.
result Sine activation functions allow for constant-width network realizations of modular addition, whereas ReLU networks require linear width scaling.
The availability of large microarray data has led to a growing interest in biclustering methods in the past decade. Several algorithms have been proposed to identify subsets of genes and conditions according to different similarity measures and under varying constraints. In this paper we focus on the exclusive row bicl…
Chaos in cerebellar cells enhances complexity of neural patterns.
problem Understanding how cerebellar granular layer represents complex information.
method Constructed a model of cerebellar granular layer with gap junctions, evaluated using reservoir computing.
result Chaotic dynamics in the cerebellar granular layer produce complex and diverse output patterns.
Implicit models can match or exceed explicit models with more test-time compute.
problem Understanding the expressive power and scaling of implicit models.
method Nonparametric analysis of expressive power, mathematical characterization of implicit operators, and test-time scaling experiments.
result Implicit models can progressively express more complex mappings through iteration, matching a richer function class with test-time compute.
We present a novel tractable generative model that extends Sum-Product Networks (SPNs) and significantly boosts their power. We call it Sum-Product-Quotient Networks (SPQNs), whose core concept is to incorporate conditional distributions into the model by direct computation using quotient nodes, e.g. $P(A|B) = \frac{P(…
The regulatory process of Drosophila is thoroughly studied for understanding a great variety of biological principles. While pattern-forming gene networks are analysed in the transcription step, post-transcriptional events (e.g. translation, protein processing) play an important role in establishing protein expression …
In this work, we propose a new family of generative flows on an augmented data space, with an aim to improve expressivity without drastically increasing the computational cost of sampling and evaluation of a lower bound on the likelihood. Theoretically, we prove the proposed flow can approximate a Hamiltonian ODE as a …
We study spectral gaps of cellular differentials for finite cyclic coverings of knot complements. Their asymptotics can be expressed in terms of irrationality exponents associated with ratios of logarithms of algebraic numbers determined by the first two Alexander polynomials. From this point of view it is natural to s…
The paper explores how simplicity leads to better out-of-distribution generalization in models.
problem Understanding the theoretical principles behind out-of-distribution (OOD) generalization in modern models.
method Examining diffusion models in image generation to analyze compositional generalization abilities and develop a theoretical framework for simplicity-based OOD generalization.
result The true, generalizable model corresponds to the simplest among consistent models, and this simplicity can be quantified and used to establish sample complexity guarantees.
Derandomizing PAC-Bayes bounds for smooth loss functions
problem Derandomizing PAC-Bayes bounds for smooth loss functions
method Exploiting smoothness properties of both the loss and the predictor class
result Bounds for deterministic predictors that involve flatness quantities
Study shows gap between uniform convergence and test error in random feature models.
problem Understanding the gap between uniform convergence and test error in random feature models.
method Analytical expressions for uniform convergence over norm balls, interpolators, and minimum norm interpolator risk derived and proved.
result Uniform convergence over interpolators still gives a non-trivial bound of test error even when classical uniform convergence is vacuous.
This paper identifies drift Lipschitz budget K as key to diffusion policy expressivity and statistical trade-offs.
problem Understanding and maximizing the expressivity of diffusion policies while managing statistical limitations.
method Identifying drift Lipschitz budget K as central, quantifying expressivity and statistical behavior, proving lower bounds, and providing practical implementation guidelines.
result Balancing expressivity and statistical complexity yields a finite-sample performance gap, with rates depending on sample size and drift type.
LoRA enhances model adaptability without increasing parameters.
problem Theoretical understanding of LoRA's adaptability.
method Theoretical analysis of LoRA's expressive power.
result LoRA can adapt any model to a smaller target model with a specific rank.
RedEx improves neural network optimization with convex optimization guarantees.
problem Difficult optimization of neural networks.
method RedEx architecture using convex optimization with semi-definite constraints.
result RedEx can efficiently learn functions fixed methods cannot.
We investigate statistical uncertainty quantification for reinforcement learning (RL) and its implications in exploration policy. Despite ever-growing literature on RL applications, fundamental questions about inference and error quantification, such as large-sample behaviors, appear to remain quite open. In this paper…
Algorithm recovers sparse PCA support from incomplete data.
problem Sparse PCA with incomplete and noisy data.
method Semidefinite program (SDP) relaxation of non-convex l1-regularized PCA. result SDP enables exact recovery of true support of sparse leading eigenvector.
Persistent homology enhances graph classification by capturing long-range graph properties.
problem Lack of formal assessment of persistent homology in graph learning.
method Brief introduction and theoretical discussion of persistent homology in graph context, followed by empirical analysis.
result Persistent homology improves graph classification, especially for data with prominent topological structures.
We demonstrate that almost all non-parametric dimensionality reduction methods can be expressed by a simple procedure: regularized loss minimization plus singular value truncation. By distinguishing the role of the loss and regularizer in such a process, we recover a factored perspective that reveals some gaps in the c…
It is well-known that the expressivity of a neural network depends on its architecture, with deeper networks expressing more complex functions. In the case of networks that compute piecewise linear functions, such as those with ReLU activation, the number of distinct linear regions is a natural measure of expressivity.…
RNNs struggle with in-context retrieval, while Transformers excel.
problem In-context retrieval capability of RNNs.
method Theoretical analysis and experimental techniques (CoT, RAG, Transformer layer).
result Enhancing RNNs with techniques improves their in-context retrieval capability, closing the representation gap with Transformers.
This paper derives radial fields on manifolds of symmetric positive definite matrices.
problem Lack of an expression for radial fields on manifolds of symmetric positive definite matrices.
method Derives an expression for radial fields on manifolds of symmetric positive definite matrices.
result Derives an expression for radial fields on manifolds of symmetric positive definite matrices.
Improved Mamba model for long-range sequence tasks.
problem Mamba's poor performance on long-range sequential tasks.
method Proposed B2S6, combining block-wise selective dynamics and channel-specific bias.
result Empirically, B2S6 outperforms S4 and S4D on LRA tasks.
SMERF improves distance learning with decision forests.
problem Subpar inference and prediction due to poor distances.
method Decision forest algorithm for distance learning.
result Empirically demonstrates ability to approximate arbitrary distances and identify features.
New approach connects quantum phases to VQA trainability, enabling better scaling.
problem Scalability issues in VQAs, especially barren plateaus.
method Analog VQA ansätze composed of quenches of a disordered Ising chain, tuning disorder strength.
result Thermalized and MBL phases reach maximal expressivity at large M, but barren plateaus emerge at smaller M in the thermalized phase. SVB method provides scalable Bayesian proportional hazards model for high-dimensional gene expression data.
problem Bayesian methods for high-dimensional sparse survival data often sacrifice uncertainty quantification or computational scalability.
method Mean-field variational approximation for scalable Bayesian proportional hazards model.
result SVB method offers posterior distribution for parameters and variable selection via posterior inclusion probabilities.
This work explores test-time scaling strategies for LLMs, improving sample efficiency and expressiveness.
problem Understanding the sample efficiency and expressiveness of test-time scaling strategies for LLMs.
method Established separation and expressiveness results for self-consistency, best-of-n, and self-correction strategies. result Self-correction enables Transformers to simulate online learning over multiple tasks without prior knowledge.
The paper calculates mass and volume of Einstein metrics in four dimensions.
problem Calculating mass and volume of Einstein metrics in four dimensions.
method Using Green's function and conformal laplacian, the paper expresses ADM mass as an integral and proves a mass-volume inequality.
result Proves a lower bound for the mass of a metric in terms of its volume, and various mass gap theorems.
BaCaDI discovers causal structures from unknown interventions.
problem Inferring causal structures from unknown interventions with limited data.
method Bayesian framework with gradient-based variational inference.
result BaCaDI outperforms related methods in identifying causal structures and intervention targets.
This paper addresses the problem of distributed detection in multi-agent networks. Agents receive private signals about an unknown state of the world. The underlying state is globally identifiable, yet informative signals may be dispersed throughout the network. Using an optimization-based framework, we develop an iter…
New technique reduces language biases in large language models.
problem Gap between language and thought in large language models.
method Proposes Language-of-Thoughts (LoT) technique to adjust order and tokens.
result Significantly reduces language biases and improves reasoning tasks.
SurVAE Flows combine VAEs and flows using surjective transformations.
problem Combining the strengths of VAEs and flows to model complex densities.
method Modular framework of composable deterministic and stochastic transformations.
result Exact likelihood computation and lower bound on likelihood.
Personalization of cardiac models involves the optimization of organ tissue properties that vary spatially over the non-Euclidean geometry model of the heart. To represent the high-dimensional (HD) unknown of tissue properties, most existing works rely on a low-dimensional (LD) partitioning of the geometrical model. Wh…
We develop a dual-control method for approximating investment strategies in incomplete environments that emerge from the presence of trading constraints. Convex duality enables the approximate technology to generate lower and upper bounds on the optimal value function. The mechanism rests on closed-form expressions per…
A new method learns hierarchical EBM models with diffusion schemes.
problem Challenges in learning EBM models with multi-modal distributions.
method Proposes a diffusion probabilistic scheme to learn EBM models in hierarchical latent spaces.
result Demonstrates superior performance on various tasks with diffusion-learned EBM.
This work improves adversarial robustness in sparse coding models.
problem The gap between theoretical models and practical deployment in adversarial robustness.
method Combining a sparsity-promoting encoder with a linear classifier, and providing a robustness certificate.
result Bounding the robust risk and providing a robustness certificate for end-to-end classification.
Sentiment analysis is a highly subjective and challenging task. Its complexity further increases when applied to the Arabic language, mainly because of the large variety of dialects that are unstandardized and widely used in the Web, especially in social media. While many datasets have been released to train sentiment …
Derives bounds for deterministic predictors using smooth loss functions.
problem Generalizing probabilistic predictors to deterministic ones.
method Exploits smoothness properties of loss and predictor classes, controlling the Jensen gap class through Rademacher complexity.
result Derives bounds for deterministic predictors involving flatness quantities from Jacobians and Hessians.
Paper explains why robust generalization is hard in deep learning models.
problem Difficulty in achieving robust generalization despite good training accuracy.
method Theoretical analysis of expressive power for deep neural networks.
result Expressive power of neural networks affects robust generalization.
Advanced GNNs improve molecular generation models.
problem Generating complete graphs with multiple nodes and edges based on labels.
method Replaced standard GNNs with more expressive GNNs in autoregressive and one-shot generation models.
result Advanced GNNs can improve performance of graph generative models, but expressiveness is not a necessity.
This work closes the gap between theory and practice for nICA identifiability.
problem Identifying latent components in nonlinearly mixed data.
method Finite-sample analysis of GCL-based nICA, combining GCL properties, statistical generalization, and numerical differentiation.
result Establishes a trade-off between function learner complexity and expressiveness.
Develops a three-currency HJM framework for Brazilian credit markets, finding significant credit spread differences between indexed segments.
problem Identifies and quantifies differences in corporate credit spreads between two parallel segments of the Brazilian bond market.
method Uses a Heath-Jarrow-Morton framework to model corporate credit as a separate economy, linking it to nominal and real economies through synthetic rates.
result Empirically finds a 640 basis point average difference in credit spreads between CDI-indexed and IPCA-indexed segments, stable through market cycles.
New method improves statistical interpolation for analyzing complex random structures.
problem Analyzing atypical random structures in statistical models.
method Introduces a large deviation upgrade to fully lifted interpolation.
result Allows for easier analysis of atypical random structures.