Deep neural networks can interpolate any dataset in the overparametrized regime.
problem Interpolating any dataset with deep neural networks in the overparametrized regime.
method Proving universal approximations and interpolating any dataset with deep neural networks, considering specific conditions on activation functions.
result Interpolation of any dataset is possible in the overparametrized regime with deep neural networks.
Unified formula for training dynamics of linear networks combining lazy and balanced regimes.
problem Training dynamics of linear networks in two distinct setups: lazy and balanced/active.
method Unified formula for the evolution of the learned matrix, combining lazy and balanced regimes.
result Unified formula allows for rapid convergence and low rank bias, proving a complete phase diagram.
Study on rich regime training in deep learning, finding active parameters in bottom layers.
problem Understanding the practical success of deep learning models.
method Empirical study on rich regime training with benchmark datasets, re-initialization analysis, and probabilistic Layer-Wise Sparse SGD.
result Probabilistic Layer-Wise Sparse SGD matches vanilla SGD's generalization performance with improved efficiency.
New insights into neural network forgetting reveal a trade-off between node activation and re-use.
problem Challenges in maintaining performance on old tasks while learning new ones.
method Theoretical analysis of synthetic and real data setups, focusing on node activation vs re-use.
result Worst forgetting occurs in an intermediate similarity regime between learned tasks.
Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.
problem Training two-layer ReLU networks using gradient descent in the mean field regime.
method Describes a condition for convergence to minimum Bayes risk, extending previous results to ReLU-activated networks.
result The condition for convergence does not depend on initialization and concerns weak convergence of network realization.
This study compares MLPs and KANs in low-data regimes, finding MLPs with personalized activation functions outperform KANs.
problem Comparing MLPs and KANs in low-data regimes.
method Introduced an effective technique for designing MLPs with unique, parameterized activation functions for each neuron.
result MLPs with personalized activation functions achieve significantly higher predictive accuracy with only a modest increase in parameters, especially in low-data regimes.
This work establishes distribution-free upper and lower bounds on the minimax label complexity of active learning with general hypothesis classes, under various noise models. The results reveal a number of surprising facts. In particular, under the noise model of Tsybakov (2004), the minimax label complexity of active …
Gradient descent dynamics in neural networks show quenching and activation phases.
problem Understanding training dynamics in neural networks.
method Numerical and phenomenological study of gradient descent algorithm for two-layer neural networks.
result Gradient descent dynamics exhibit quenching and activation phases in under-parametrized networks.
Neural networks encode inputs deterministically and categorically, behaving like hash encoders.
problem Understanding the encoding properties of neural networks.
method Analyzed the input space partitioned by ReLU-like activations in neural networks.
result Neural networks can be represented by unique activation patterns, similar to hash encoders.
Model shows loss curve with two distinct exponents due to sparse activations.
problem Sparse activations impact neural network scaling laws.
method Introduced a model for neural scaling laws under sparse activations, derived asymptotic population loss, and analyzed gradient-descent dynamics.
result Loss curve exhibits double-descent peak near interpolation threshold with two distinct scaling exponents.
New algorithm learns switching dynamics from multiple neural signals.
problem Learning accurate switching dynamical system models from multimodal neural data.
method Unsupervised learning algorithm for multiscale switching dynamical system models.
result Switching multiscale dynamical system models outperform single-scale models in behavior decoding.
Model shows stock markets can be inefficiently mispriced.
problem Limits of informationally efficient stock markets.
method Chartist-fundamentalist model with chartists and fundamentalists trading conditions.
result Stock markets can exhibit constant or oscillatory mispricing.
Deep learning has become a powerful and popular tool for a variety of machine learning tasks. However, it is challenging to understand the mechanism of deep learning from a theoretical perspective. In this work, we propose a random active path model to study collective properties of deep neural networks with binary syn…
Sine activation functions enable two-layer neural networks to learn modular addition more efficiently.
problem Learning modular addition with two-layer neural networks.
method Introduced and analyzed sine activation functions, providing theoretical and empirical evidence.
result Sine activation functions allow for constant-width network realizations of modular addition, whereas ReLU networks require linear width scaling.
Study on critical points in random neural networks, revealing three regimes based on activation function.
problem Investigating the expected number of critical points in random neural networks.
method Deriving asymptotic formulas for critical points under infinite-width limit and suitable regularity conditions.
result Three distinct regimes of critical points behavior depending on activation function.
New framework improves fairness in small data settings.
problem Ensuring fairness in low-data environments.
method Combines posterior sampling exploration with fair classification.
result Framework maximizes accuracy while meeting fairness constraints.
A new method for active learning works well across all label budgets.
problem Active learning methods perform poorly in both low and high label budgets.
method Uncertainty Herding: a simple, computationally fast method that optimizes uncertainty coverage.
result Uncertainty Herding nearly optimizes distribution-level coverage and performs well across various active learning tasks.
The paper studies how neural networks evolve representations, finding a unique fixed point for nonlinear activations.
problem Understanding how neural networks transform input data across layers.
method Theoretical framework for the evolution of the kernel sequence, using mean-field regime and Hermite polynomials.
result For nonlinear activations, the kernel sequence converges globally to a unique fixed point.
We introduce 'semi-unsupervised learning', a problem regime related to transfer learning and zero-shot learning where, in the training data, some classes are sparsely labelled and others entirely unlabelled. Models able to learn from training data of this type are potentially of great use as many real-world datasets ar…
Recent work by Jacot et al. (2018) has shown that training a neural network using gradient descent in parameter space is related to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK). Lee et al. (2019) built on this result by establishing that the output of a neural network traine…
Study long-only minimum variance portfolio in one-factor market with arbitrary sign betas.
problem Characterize the long-only minimum variance portfolio in a one-factor market with mixed-sign betas.
method Explicit solution for long-only minimum variance portfolio, explicit characterization of active set, asymptotic analysis in high-dimensional regime.
result Proportion of active assets in LOMV portfolio converges to F(β∗) in high-dimensional regime, with rate O(F(0)1/3) when F(0)>0. The article uses dynamic factor allocation to improve portfolio performance by integrating regime-switching signals.
problem Improving portfolio performance through dynamic factor allocation.
method The authors apply the sparse jump model (SJM) to identify bull and bear market regimes for individual factors, then fine-tune hyperparameters using a hypothetical single-factor long-short strategy. These regime inferences are incorporated into the Black-Litterman framework to dynamically adjust allocations among indices.
result The constructed multi-factor portfolio significantly improves the information ratio (IR) relative to the market, raising it from 0.05 to approximately 0.4.
MRC improves credit assignment in multi-agent LLM systems, achieving high returns and transparency.
problem Lack of principled credit assignment in multi-agent LLM decision systems, vulnerability to regime shifts, and limited transparency.
method Market Regime Council (MRC) computes exact Shapley credits, uses exponentially weighted performance histories, Bayesian adaptive mixture, and regime-dependent multipliers.
result MRC achieves a Sharpe ratio of 1.51 and a cumulative return of 440.1% over 1,037 trading days, ranking first on CR, SR, and IR.
Unified framework for hybrid learning and optimization via active inference.
problem Sequential decisions in black-box evaluations requiring both task improvement and uncertainty reduction.
method Pragmatic Curiosity (PraC) framework that evaluates queries by balancing information gain and pragmatic value.
result Unified approach reduces decision risk and improves coverage of critical regions without task-specific rules.
New method reduces labeling costs in semi-supervised learning.
problem Efficiently label data with limited labeled samples.
method Formulates batch acquisition as bilevel optimization.
result Shows effectiveness in keyword detection tasks.
Study finds meme stocks have unique price and social media dynamics.
problem Exploring unique properties of meme stocks.
method Regime-switching cointegration model.
result Meme stocks exhibit a distinct 'mementum' compared to other high-volume stocks.
New insights into neural network training efficiency.
problem Understanding the optimal initialization for deep neural networks.
method Exploring the edge of chaos and saturation of tanh activation function.
result The line of uniformity in phase space intersects the edge of chaos, indicating saturation begins to hinder training efficiency.
In this paper, we analyze efficacy of the fast gradient sign method (FGSM) and the Carlini-Wagner's L2 (CW-L2) attack. We prove that, within a certain regime, the untargeted FGSM can fool any convolutional neural nets (CNNs) with ReLU activation; the targeted FGSM can mislead any CNNs with ReLU activation to classify a…
Study risk-constrained Kelly optimization for mutually exclusive outcomes, proving support invariance and developing a structured algorithm.
problem Risk-constrained Kelly optimization for mutually exclusive outcomes with explicit state prices.
method Analyzes the finite mutually exclusive outcome version of risk-constrained Kelly optimization with explicit state prices, proving support invariance and developing a structured algorithm.
result Support is invariant across CRRA parameter and drawdown-surrogate parameter in the overround regime.
Study on how initialization scale affects neural network training regimes.
problem Understanding the transition between kernel and rich regimes in overparametrized models.
method Analysis of simple depth-D models and empirical testing on complex models.
result Scale of initialization controls transition between kernel and rich regimes.
The paper explains the richness scale of wide neural networks.
problem Understanding the behavior of overparameterized neural networks.
method Nonrigorous derivation and empirical evidence.
result Wide neural networks exhibit a richness scale from lazy kernel behavior to feature learning.
A recent line of work studies overparametrized neural networks in the "kernel regime," i.e. when the network behaves during training as a kernelized linear predictor, and thus training with gradient descent has the effect of finding the minimum RKHS norm solution. This stands in contrast to other studies which demonstr…
Batch Normalization (BN) uses mini-batch statistics to normalize the activations during training, introducing dependence between mini-batch elements. This dependency can hurt the performance if the mini-batch size is too small, or if the elements are correlated. Several alternatives, such as Batch Renormalization and G…
New method uses neural tangent kernel for efficient active learning.
problem Efficiently approximating deep learning's look-ahead selection criteria.
method Approximates retraining with neural tangent kernel for active learning.
result Approximation works asymptotically and enables sequential active learning.
GRIP2 improves deep learning feature selection robustness in correlated and noisy data.
problem Identifying predictive features in correlated and noisy data.
method Integrates first-layer feature activity over a two-dimensional regularization surface to control sparsity and geometry, using efficient block-stochastic sampling.
result Demonstrates improved robustness and power in high correlation and low signal-to-noise ratio regimes.
Study of two-layer ReLU neural network phase diagram at infinite-width limit.
problem Characterize the dynamical regimes of two-layer ReLU neural networks.
method Combining experimental and theoretical approaches, including phase diagram analogy.
result Identification of three regimes: linear, critical, and condensed.
New method identifies nonstationary causal structures in time series data.
problem Identifying causal relationships in time series data that change over time.
method High-order Markov Switching Models for regime-dependent causal discovery.
result Scalable approach for estimating high-order regime-dependent causal structures.
This paper introduces a novel model-based clustering approach for clustering time series which present changes in regime. It consists of a mixture of polynomial regressions governed by hidden Markov chains. The underlying hidden process for each cluster activates successively several polynomial regimes during time. The…
Active Kriging Monte Carlo simulation method with conformal certification for failure probability estimation
problem Failure probability estimation in structural reliability analysis
method Active learning framework with conformal prediction
result Improved uncertainty quantification and reliability of failure probability estimates
Wide neural networks can be closely approximated by Gaussian processes, with rates depending on the activation function's properties.
problem Approximating the behavior of wide neural networks using Gaussian processes.
method Established convergence rates for the central limit theorem in an infinite-dimensional functional space, using a transportation distance metric.
result Explicit convergence rates for neural networks approximated by Gaussian processes, varying based on the activation function's properties.
Depth uncertainty networks don't improve with bias correction, contrary to expectations.
problem Improving performance in active learning with overparameterised models like NNs.
method Depth uncertainty networks, compared to underparameterised models, show no improvement in performance with bias correction.
result Depth uncertainty networks do not improve with bias correction, unlike underparameterised models.
Temporal Functional Circuits explain KAN forecasts with interpretable edge functions.
problem Lack of mechanistic explanations in KAN forecasting.
method Transform KAN edge functions into faithful, temporally grounded explanations using a gated residual KAN.
result Gated KAN achieves lower MSE than linear-only models on regime-switching signals.
In the world of big data, large but costly to label datasets dominate many fields. Active learning, a semi-supervised alternative to the standard PAC-learning model, was introduced to explore whether adaptive labeling could learn concepts with exponentially fewer labeled samples. While previous results show that active…
We study the competitive equilibrium of large random economies with linear activities using methods of statistical mechanics. We focus on economies with C commodities, N firms, each running a randomly drawn linear technology, and one consumer. We derive, in the limit N,C→∞ with n=N/C fixed, a complete de…
Dual neural network architecture improves accuracy and interpretability.
problem Improving neural network interpretability and accuracy.
method Stacked recurrent and feedforward layers, binary activation function.
result Binary activation leads to simpler, more interpretable models with higher accuracy.
Improves active learning efficiency by warping input space based on observed outputs.
problem Insensitivity of Gaussian process uncertainty to actual observations.
method Input warping with learned monotone reparameterization to adjust acquisition function behavior.
result Significantly improved sample efficiency across various benchmarks, especially in non-stationary conditions.
New method detects and clusters market regimes in multidimensional data.
problem Detecting and clustering market regimes in complex data structures.
method Non-parametric online market regime detection and clustering using path-wise two-sample tests and maximum mean discrepancy.
result Successfully detected and clustered market regimes in various data structures.
The paper investigates how activation functions impact the training of Neural ODEs, leading to global convergence.
problem Challenges in training Neural ODEs, particularly gradient computation accuracy and convergence analysis.
method Investigates the impact of activation functions on the training dynamics of Neural ODEs.
result Establishes global convergence of Neural ODEs under gradient descent in overparameterized regimes.