Looped transformers outperform standard transformers in complex reasoning tasks due to a specific loss landscape geometry.
problem Understanding why looped transformers outperform standard transformers in complex reasoning tasks.
method Explained through loss landscape geometry, distinguishing between U-shaped and V-shaped valleys, and proposing SHIFT training strategy.
result Looped transformers' recursive architecture induces a River-V-Valley landscape, leading to better loss convergence and complex pattern learning.
We present novel empirical observations regarding how stochastic gradient descent (SGD) navigates the loss landscape of over-parametrized deep neural networks (DNNs). These observations expose the qualitatively different roles of learning rate and batch-size in DNN optimization and generalization. Specifically we study…
Neural networks provide a rich class of high-dimensional, non-convex optimization problems. Despite their non-convexity, gradient-descent methods often successfully optimize these models. This has motivated a recent spur in research attempting to characterize properties of their loss surface that may explain such succe…
This paper explores loss landscapes of sparse neural networks, finding unique characteristics compared to dense networks.
problem Understanding the loss landscape of sparse neural networks, especially one-hidden-layer networks.
method Analyzes sparse networks with dense and sparse final layers, focusing on linear and non-linear models.
result Sparse networks can have no spurious valleys under certain conditions, but spurious valleys and minima can exist for wide sparse networks.
This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large…
Despite the non-convex nature of their loss functions, deep neural networks are known to generalize well when optimized with stochastic gradient descent (SGD). Recent work conjectures that SGD with proper configuration is able to find wide and flat local minima, which have been proposed to be associated with good gener…
LoRA-Curve connects independent LoRA optima through continuous low-loss valleys, improving Bayesian model averaging.
problem Challenges in estimating epistemic uncertainty in LoRA-based Bayesian inference.
method Introduces LoRA-Curve, a segmented Bézier curve parameterization in the LoRA space, with free and anchored configurations.
result Empirically shows that connecting independent LoRA optima through continuous low-loss valleys improves mutual information of the predictive distribution.
A new pruning method reduces neural network computation without retraining.
problem Efficiently reduce neural network computation while maintaining accuracy.
method Structured directional pruning via perturbation orthogonal projection.
result Achieves state-of-the-art pruned accuracy without retraining.
A D-Wave quantum annealer (QA) having a 2048 qubit lattice, with no missing qubits and couplings, allowed embedding of a complete graph of a Restricted Boltzmann Machine (RBM). A handwritten digit OptDigits data set having 8x7 pixels of visible units was used to train the RBM using a classical Contrastive Divergence. E…
A new Kolmogorov-Arnold network improves function approximation and optimization.
problem Approximating potentially irregular functions in high dimensions.
method Proposes a new Kolmogorov-Arnold network (KAN) and provides error bounds and universal approximation theorems.
result Outperforms multilayer perceptrons in accuracy and convergence speed for irregular functions.
A common difficulty in applications of machine learning is the lack of any general principle for guiding the choices of key parameters of the underlying neural network. Focusing on a class of recurrent neural networks - reservoir computing systems that have recently been exploited for model-free prediction of nonlinear…
We identify a class of over-parameterized deep neural networks with standard activation functions and cross-entropy loss which provably have no bad local valley, in the sense that from any point in parameter space there exists a continuous path on which the cross-entropy loss is non-increasing and gets arbitrarily clos…
This paper examines SVB's failure and its impact on bank stocks.
problem SVB failure and its contagion effects on bank stocks.
method Analyzed bank-specific vulnerabilities and stock performance.
result Uninsured deposits and unrealized losses were key factors in SVB's impact.
Proposes a method to partition univariate data into unimodal subsets.
problem Partitioning univariate multimodal data into unimodal subsets.
method Recursive splitting around valley points of the data density using properties of critical points on the convex hull of the ecdf plot.
result Obtains a hierarchical statistical model of the initial dataset as a mixture of UMMs.
New framework reveals thermodynamic principles for LLM training.
problem Understanding the training dynamics of large language models.
method Introducing Neural Thermodynamic Laws (NTL) under river-valley loss landscape assumptions.
result Key thermodynamic quantities and principles naturally emerge in LLM training.
WSD schedule improves model training efficiency by adapting learning rates dynamically.
problem Fixed compute budgets limit training efficiency of language models.
method Introduces a WSD schedule that uses a constant learning rate followed by a rapid decay phase.
result WSD schedule generates a non-traditional loss curve with stable and decay phases.
HyPV-LEAD detects cryptocurrency anomalies proactively, improving financial security.
problem Cryptocurrency anomalies like mixing, fraud, and pump-and-dump operations are hard to detect due to class imbalance and temporal volatility.
method HyPV-LEAD integrates lead time into anomaly detection through window-horizon modeling, Peak-Valley sampling, and hyperbolic embedding.
result HyPV-LEAD achieves a PR-AUC of 0.9624 on Bitcoin transaction data, significantly outperforming state-of-the-art methods.
Quantization-aware training can recover accuracy lost by post-training quantization.
problem Post-training quantization (PTQ) can fail sharply at aggressive bitwidths.
method A unified geometric framework that explains PTQ failure and QAT recovery.
result QAT has a useful bias that steers iterates back into the basin.
This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenval…
Solves local minima problems on smooth manifolds.
problem Local minima issues on smooth manifolds.
method Introducing valley functions and applying Morse's lemma.
result Eliminates critical points and reduces to 1D.
Unbalanced data arises in many learning tasks such as clustering of multi-class data, hierarchical divisive clustering and semisupervised learning. Graph-based approaches are popular tools for these problems. Graph construction is an important aspect of graph-based learning. We show that graph-based algorithms can fail…
Designs chiral photonic structures using machine learning for efficient optical properties.
problem Optimizing chiral photonic nanostructures for light-matter interactions.
method Evolutionary algorithm and neural network approach for rapid optimization.
result Frequency-dependent modification in reflected light's degree of circular polarization.
Tilting loss functions improves machine learning performance.
problem Improving machine learning models, especially in under- and over-parameterized networks.
method Using evolving loss functions that emphasize different classes cyclically.
result Dynamical loss functions lead to better generalization and stability in training.
Stablecoin liquidity was affected by the SVB collapse, with USDC's transparency leading to market reactions.
problem Impact of stablecoin transparency on liquidity during market turmoil.
method Adapted MCI measure to Uniswap, Difference-in-Differences analysis on MCI and TVL, measured liquidity concentration.
result USDC's transparency led to swift market reactions, while USDT's opacity provided a safety net.
Large SGD step sizes lead to sparse feature learning in neural networks.
problem Sparse feature learning in neural networks with large step sizes.
method Empirical observations and theoretical analysis of SGD dynamics.
result Large step sizes induce implicit regularization leading to sparse predictors.
A new pruning method finds sparse minimizers in flat regions of deep neural networks.
problem Finding sparse minimizers in flat regions of deep neural networks.
method Directional pruning using an ℓ1 proximal gradient algorithm. result Empirical results show promising sparsity (92%) and comparable minima to SGD.
Adaptor 'E' extends gradient-based optimizers to explore loss landscapes, improving generalization.
problem Finding lower and better-generalizing minima in deep learning.
method Proposes an adaptor 'E' to extend gradient-based optimizers, encouraging exploration along landscape valleys.
result Adapted optimizers increase test accuracy by an average of 2.5% in large-batch training tasks.
Cryptos remained resilient after SVB's collapse, contrary to expectations.
problem Impact of SVB collapse on crypto markets.
method Factual summary, sentiment analysis, and market performance examination.
result Cryptocurrencies showed resilience after SVB's collapse.
Piecewise linear activations create many spurious local minima in neural networks.
problem Understanding the loss surface of neural networks with piecewise linear activations.
method Proved the existence of infinite spurious local minima and partitioned the loss surface into smooth cells.
result Piecewise linear activations create many spurious local minima that are invariant under a continuous path.
Model shows how banks' hidden-to-maturity accounting can mask run risk and lead to financial instability.
problem Run risk and hidden-to-maturity accounting in banking systems.
method Balance sheet model and optimization problem to assess run risk and resilience.
result Held-to-maturity accounting can mask revaluation losses and increase run risk.
D-Wave quantum annealing fails to improve sampling quality from RBMs compared to Gibbs sampling.
problem Improving sampling quality from RBMs using D-Wave quantum annealing.
method Comparison of D-Wave quantum annealing and Gibbs sampling for RBM sampling.
result D-Wave sampling does not significantly improve the number of local valleys compared to Gibbs sampling.
SALR improves deep learning generalization by dynamically adjusting learning rates.
problem Improving generalization in deep learning models.
method Sharpness-aware learning rate scheduling based on local loss function sharpness.
result SALR drives solutions to flatter regions, improving generalization and convergence.
We study the phenomenon that some modules of deep neural networks (DNNs) are more critical than others. Meaning that rewinding their parameter values back to initialization, while keeping other modules fixed at the trained parameters, results in a large drop in the network's performance. Our analysis reveals interestin…
Study the landscape of Lipschitz functions between manifolds using persistent homology.
problem Understanding the structure of homotopy paths between maps with high Lipschitz constants.
method Using persistent homology to analyze the landscape of Lipschitz functions between manifolds.
result First results on the persistence of higher-dimensional cycles in function spaces.
The permutation symmetry of neurons in each layer of a deep neural network gives rise not only to multiple equivalent global minima of the loss function, but also to first-order saddle points located on the path between the global minima. In a network of d−1 hidden layers with nk neurons in layers $k = 1, \ldots, …
Stochastic gradient descent (SGD) forms the core optimization method for deep neural networks. While some theoretical progress has been made, it still remains unclear why SGD leads the learning dynamics in overparameterized networks to solutions that generalize well. Here we show that for overparameterized networks wit…
Model shows how capital accumulation can lead to poverty traps and well-being states.
problem Capital accumulation and its effects on poverty and well-being.
method Stochastic Solow growth model with sigmoidal saving fraction and bimodal steady state distribution.
result Existence of poverty trap with fluctuation-driven transitions between poverty and well-being states.
New algorithm ATENT improves adversarial robustness in neural networks.
problem Improving neural network robustness against adversarial attacks.
method Proposes a new loss function with entropic regularization for training robust neural networks.
result ATENT achieves competitive robust classification accuracy on benchmark datasets.
We build a model using Gaussian processes to infer a spatio-temporal vector field from observed agent trajectories. Significant landmarks or influence points in agent surroundings are jointly derived through vector calculus operations that indicate presence of sources and sinks. We evaluate these influence points by us…
A new AI optimization method uses energy-conserving dynamics inspired by Born-Infeld theory.
problem Optimization challenges in non-convex loss functions and machine learning tasks.
method Discretization of Born-Infeld dynamics for energy-conserving Hamiltonian optimization.
result The method avoids high local minima and outperforms traditional methods in shallow valleys.
Randomized algorithms that base iteration-level decisions on samples from some pool are ubiquitous in machine learning and optimization. Examples include stochastic gradient descent and randomized coordinate descent. This paper makes progress at theoretically evaluating the difference in performance between sampling wi…
This paper presents an algorithm for a complete and efficient calibration of the Heston stochastic volatility model. We express the calibration as a nonlinear least squares problem. We exploit a suitable representation of the Heston characteristic function and modify it to avoid discontinuities caused by branch switchi…
DiMS sampler explores neural network loss minima via dissipative dynamics.
problem Sampling reparameterization invariant solutions in neural networks.
method Dynamical system based on kinetic energy with dissipative friction.
result DiMS sampler samples exactly from minimum level sets.
Combining insights from machine learning and quantum Monte Carlo, the stochastic reconfiguration method with neural network Ansatz states is a promising new direction for high-precision ground state estimation of quantum many-body problems. Even though this method works well in practice, little is known about the learn…
Data analysis in high-dimensional spaces aims at obtaining a synthetic description of a data set, revealing its main structure and its salient features. We here introduce an approach providing this description in the form of a topography of the data, namely a human-readable chart of the probability density from which t…
Groups of firms often achieve a competitive advantage through the formation of geo-industrial clusters. Although many exemplary clusters, such as Hollywood or Silicon Valley, have been frequently studied, systematic approaches to identify and analyze the hierarchical structure of the geo-industrial clusters at the glob…
Study identifies new stable climate states in climate model.
problem Understanding multistability and transitions in climate models.
method Combination of quasipotential theory and manifold learning.
result Discovery of a third stable climate state not previously known.
Bayesian model averaging (BMA) is the state of the art approach for overcoming model uncertainty. Yet, especially on small data sets, the results yielded by BMA might be sensitive to the prior over the models. Credal Model Averaging (CMA) addresses this problem by substituting the single prior over the models by a set …