LoRA-Curve connects independent LoRA optima through continuous low-loss valleys, improving Bayesian model averaging.
problem Challenges in estimating epistemic uncertainty in LoRA-based Bayesian inference.
method Introduces LoRA-Curve, a segmented Bézier curve parameterization in the LoRA space, with free and anchored configurations.
result Empirically shows that connecting independent LoRA optima through continuous low-loss valleys improves mutual information of the predictive distribution.
Quantization-aware training can recover accuracy lost by post-training quantization.
problem Post-training quantization (PTQ) can fail sharply at aggressive bitwidths.
method A unified geometric framework that explains PTQ failure and QAT recovery.
result QAT has a useful bias that steers iterates back into the basin.
DiMS sampler explores neural network loss minima via dissipative dynamics.
problem Sampling reparameterization invariant solutions in neural networks.
method Dynamical system based on kinetic energy with dissipative friction.
result DiMS sampler samples exactly from minimum level sets.
New methods connect low-loss points on neural network surfaces.
problem Connecting low-loss points on neural network loss surfaces.
method Macroscopic distributional assumptions and global connection models.
result Accuracy correlates with complexity and sensitivity.
Looped transformers outperform standard transformers in complex reasoning tasks due to a specific loss landscape geometry.
problem Understanding why looped transformers outperform standard transformers in complex reasoning tasks.
method Explained through loss landscape geometry, distinguishing between U-shaped and V-shaped valleys, and proposing SHIFT training strategy.
result Looped transformers' recursive architecture induces a River-V-Valley landscape, leading to better loss convergence and complex pattern learning.
We present novel empirical observations regarding how stochastic gradient descent (SGD) navigates the loss landscape of over-parametrized deep neural networks (DNNs). These observations expose the qualitatively different roles of learning rate and batch-size in DNN optimization and generalization. Specifically we study…
Neural networks provide a rich class of high-dimensional, non-convex optimization problems. Despite their non-convexity, gradient-descent methods often successfully optimize these models. This has motivated a recent spur in research attempting to characterize properties of their loss surface that may explain such succe…
This paper explores loss landscapes of sparse neural networks, finding unique characteristics compared to dense networks.
problem Understanding the loss landscape of sparse neural networks, especially one-hidden-layer networks.
method Analyzes sparse networks with dense and sparse final layers, focusing on linear and non-linear models.
result Sparse networks can have no spurious valleys under certain conditions, but spurious valleys and minima can exist for wide sparse networks.
New methods evaluate data representations by complexity of low-loss predictor learning.
problem Evaluating quality of data representations for downstream tasks.
method Surplus Description Length (SDL) and ε Sample Complexity (εSC) methods.
result Methods measure the information needed to approximate optimal predictor up to specified tolerance.
This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large…
Despite the non-convex nature of their loss functions, deep neural networks are known to generalize well when optimized with stochastic gradient descent (SGD). Recent work conjectures that SGD with proper configuration is able to find wide and flat local minima, which have been proposed to be associated with good gener…
A new pruning method reduces neural network computation without retraining.
problem Efficiently reduce neural network computation while maintaining accuracy.
method Structured directional pruning via perturbation orthogonal projection.
result Achieves state-of-the-art pruned accuracy without retraining.
A D-Wave quantum annealer (QA) having a 2048 qubit lattice, with no missing qubits and couplings, allowed embedding of a complete graph of a Restricted Boltzmann Machine (RBM). A handwritten digit OptDigits data set having 8x7 pixels of visible units was used to train the RBM using a classical Contrastive Divergence. E…
A new Kolmogorov-Arnold network improves function approximation and optimization.
problem Approximating potentially irregular functions in high dimensions.
method Proposes a new Kolmogorov-Arnold network (KAN) and provides error bounds and universal approximation theorems.
result Outperforms multilayer perceptrons in accuracy and convergence speed for irregular functions.
A common difficulty in applications of machine learning is the lack of any general principle for guiding the choices of key parameters of the underlying neural network. Focusing on a class of recurrent neural networks - reservoir computing systems that have recently been exploited for model-free prediction of nonlinear…
How to model and encode the semantics of human-written text and select the type of neural network to process it are not settled issues in sentiment analysis. Accuracy and transferability are critical issues in machine learning in general. These properties are closely related to the loss estimates for the trained model.…
We identify a class of over-parameterized deep neural networks with standard activation functions and cross-entropy loss which provably have no bad local valley, in the sense that from any point in parameter space there exists a continuous path on which the cross-entropy loss is non-increasing and gets arbitrarily clos…
Proposes a method to ensure low losses across all subpopulations in large datasets.
problem Standard practice of minimizing average loss fails to guarantee low losses across all subpopulations in heterogeneous datasets.
method Convex procedure that controls worst-case performance over all subpopulations of a given size with finite-sample convergence guarantees.
result Empirically, the worst-case procedure learns models that do well against unseen subpopulations.
Novel approach embeds loss tunnels in neural networks, revealing insights into their structure.
problem Understanding the structure of neural network loss surfaces, especially low-loss tunnels.
method Directly embedding loss tunnels into the loss landscape of neural networks.
result Improved insights into the length and structure of loss tunnels, and better subspace inference in Bayesian neural networks.
This paper examines SVB's failure and its impact on bank stocks.
problem SVB failure and its contagion effects on bank stocks.
method Analyzed bank-specific vulnerabilities and stock performance.
result Uninsured deposits and unrealized losses were key factors in SVB's impact.
Proposes a method to partition univariate data into unimodal subsets.
problem Partitioning univariate multimodal data into unimodal subsets.
method Recursive splitting around valley points of the data density using properties of critical points on the convex hull of the ecdf plot.
result Obtains a hierarchical statistical model of the initial dataset as a mixture of UMMs.
New framework reveals thermodynamic principles for LLM training.
problem Understanding the training dynamics of large language models.
method Introducing Neural Thermodynamic Laws (NTL) under river-valley loss landscape assumptions.
result Key thermodynamic quantities and principles naturally emerge in LLM training.
Paper discovers simplicial complexes connecting trained models for improved ensembling.
problem Improving robustness and accuracy of deep learning ensembles.
method Identifies mode-connecting simplicial complexes on loss surfaces.
result Efficiently builds simplicial complexes for ensembling, outperforming independent ensembles.
WSD schedule improves model training efficiency by adapting learning rates dynamically.
problem Fixed compute budgets limit training efficiency of language models.
method Introduces a WSD schedule that uses a constant learning rate followed by a rapid decay phase.
result WSD schedule generates a non-traditional loss curve with stable and decay phases.
HyPV-LEAD detects cryptocurrency anomalies proactively, improving financial security.
problem Cryptocurrency anomalies like mixing, fraud, and pump-and-dump operations are hard to detect due to class imbalance and temporal volatility.
method HyPV-LEAD integrates lead time into anomaly detection through window-horizon modeling, Peak-Valley sampling, and hyperbolic embedding.
result HyPV-LEAD achieves a PR-AUC of 0.9624 on Bitcoin transaction data, significantly outperforming state-of-the-art methods.
New framework for neural networks converging to low loss without overparameterization.
problem Training deep neural networks without overparameterization assumptions.
method Construction of random sparse lifts and analysis using algebraic topology and random graph theory.
result Provable convergence to low loss for large sparse neural networks.
This paper proposes a new optimization algorithm called Entropy-SGD for training deep neural networks that is motivated by the local geometry of the energy landscape. Local extrema with low generalization error have a large proportion of almost-zero eigenvalues in the Hessian with very few positive or negative eigenval…
Solves local minima problems on smooth manifolds.
problem Local minima issues on smooth manifolds.
method Introducing valley functions and applying Morse's lemma.
result Eliminates critical points and reduces to 1D.
Unbalanced data arises in many learning tasks such as clustering of multi-class data, hierarchical divisive clustering and semisupervised learning. Graph-based approaches are popular tools for these problems. Graph construction is an important aspect of graph-based learning. We show that graph-based algorithms can fail…
Geodesics connect model modes in neural network loss landscapes.
problem Connecting modes in neural network loss landscapes.
method Reframed mode connectivity in Information Geometry, hypothesized geodesics as mode-connecting paths, proposed algorithm to approximate geodesics.
result Geodesics achieve mode connectivity in neural networks.
Designs chiral photonic structures using machine learning for efficient optical properties.
problem Optimizing chiral photonic nanostructures for light-matter interactions.
method Evolutionary algorithm and neural network approach for rapid optimization.
result Frequency-dependent modification in reflected light's degree of circular polarization.
Tilting loss functions improves machine learning performance.
problem Improving machine learning models, especially in under- and over-parameterized networks.
method Using evolving loss functions that emphasize different classes cyclically.
result Dynamical loss functions lead to better generalization and stability in training.
Stablecoin liquidity was affected by the SVB collapse, with USDC's transparency leading to market reactions.
problem Impact of stablecoin transparency on liquidity during market turmoil.
method Adapted MCI measure to Uniswap, Difference-in-Differences analysis on MCI and TVL, measured liquidity concentration.
result USDC's transparency led to swift market reactions, while USDT's opacity provided a safety net.
Large SGD step sizes lead to sparse feature learning in neural networks.
problem Sparse feature learning in neural networks with large step sizes.
method Empirical observations and theoretical analysis of SGD dynamics.
result Large step sizes induce implicit regularization leading to sparse predictors.
A new pruning method finds sparse minimizers in flat regions of deep neural networks.
problem Finding sparse minimizers in flat regions of deep neural networks.
method Directional pruning using an ℓ1 proximal gradient algorithm. result Empirical results show promising sparsity (92%) and comparable minima to SGD.
Adaptor 'E' extends gradient-based optimizers to explore loss landscapes, improving generalization.
problem Finding lower and better-generalizing minima in deep learning.
method Proposes an adaptor 'E' to extend gradient-based optimizers, encouraging exploration along landscape valleys.
result Adapted optimizers increase test accuracy by an average of 2.5% in large-batch training tasks.
Cryptos remained resilient after SVB's collapse, contrary to expectations.
problem Impact of SVB collapse on crypto markets.
method Factual summary, sentiment analysis, and market performance examination.
result Cryptocurrencies showed resilience after SVB's collapse.
Piecewise linear activations create many spurious local minima in neural networks.
problem Understanding the loss surface of neural networks with piecewise linear activations.
method Proved the existence of infinite spurious local minima and partitioned the loss surface into smooth cells.
result Piecewise linear activations create many spurious local minima that are invariant under a continuous path.
Model shows how banks' hidden-to-maturity accounting can mask run risk and lead to financial instability.
problem Run risk and hidden-to-maturity accounting in banking systems.
method Balance sheet model and optimization problem to assess run risk and resilience.
result Held-to-maturity accounting can mask revaluation losses and increase run risk.
D-Wave quantum annealing fails to improve sampling quality from RBMs compared to Gibbs sampling.
problem Improving sampling quality from RBMs using D-Wave quantum annealing.
method Comparison of D-Wave quantum annealing and Gibbs sampling for RBM sampling.
result D-Wave sampling does not significantly improve the number of local valleys compared to Gibbs sampling.
Flat-minima optimizers improve neural network generalization.
problem Improving neural network generalization performance.
method Stochastic Weight Averaging (SWA) and Sharpness-Aware Minimization (SAM).
result Surprising findings from loss surface analysis and broad benchmarking.
SALR improves deep learning generalization by dynamically adjusting learning rates.
problem Improving generalization in deep learning models.
method Sharpness-aware learning rate scheduling based on local loss function sharpness.
result SALR drives solutions to flatter regions, improving generalization and convergence.
Neural networks' optimization dynamics are confined to a single basin despite connected basins in the loss landscape.
problem Neural networks' optimization dynamics are confined to a single basin despite connected basins in the loss landscape.
method Identifying entropic barriers arising from the interplay between curvature variations along low-loss paths and noise in optimization dynamics.
result Curvature-induced entropic forces bias noisy dynamics back toward the endpoints, explaining the confinement and connectivity of solutions.
Study the landscape of Lipschitz functions between manifolds using persistent homology.
problem Understanding the structure of homotopy paths between maps with high Lipschitz constants.
method Using persistent homology to analyze the landscape of Lipschitz functions between manifolds.
result First results on the persistence of higher-dimensional cycles in function spaces.
The permutation symmetry of neurons in each layer of a deep neural network gives rise not only to multiple equivalent global minima of the loss function, but also to first-order saddle points located on the path between the global minima. In a network of d−1 hidden layers with nk neurons in layers $k = 1, \ldots, …
Stochastic gradient descent (SGD) forms the core optimization method for deep neural networks. While some theoretical progress has been made, it still remains unclear why SGD leads the learning dynamics in overparameterized networks to solutions that generalize well. Here we show that for overparameterized networks wit…
Model shows how capital accumulation can lead to poverty traps and well-being states.
problem Capital accumulation and its effects on poverty and well-being.
method Stochastic Solow growth model with sigmoidal saving fraction and bimodal steady state distribution.
result Existence of poverty trap with fluctuation-driven transitions between poverty and well-being states.
New algorithm ATENT improves adversarial robustness in neural networks.
problem Improving neural network robustness against adversarial attacks.
method Proposes a new loss function with entropic regularization for training robust neural networks.
result ATENT achieves competitive robust classification accuracy on benchmark datasets.