New approach ties loss curvature to model performance in deep learning.
problem Understanding the relationship between loss curvature and model performance in deep learning.
method Empirical analysis of loss Hessians and theoretical results on input-output Jacobians.
result Novel generalization bound in terms of empirical Jacobian.
Study inverse problems with measure samples, improving estimator calibration and recovery.
problem Inverse problems with unknown potentials observed through measure samples.
method Introduced convex empirical objectives and sharpened Fenchel--Young losses for finite-dimensional potential classes.
result High-probability parameter recovery bounds for inverse entropic unbalanced optimal transport and inverse JKO learning.
Adversarial training makes logistic regression weight loss landscapes sharper.
problem Understanding why adversarial training sharpens the weight loss landscape in logistic regression.
method Theoretical analysis of linear logistic regression model with L2 norm constraints, and experiments on ResNet18.
result Adversarial training sharpens the weight loss landscape in linear logistic regression models.
It has been argued in the past that high-dimensional neural networks do not exhibit local minima capable of trapping an optimisation algorithm. However, the relationship between loss surface modality and the neural architecture parameters, such as the number of hidden neurons per layer and the number of hidden layers, …
Cohen et al. (2021) show GD trajectories align on a bifurcation diagram.
problem Understanding the Edge of Stability (EoS) phenomenon in gradient descent.
method Empirical studies and rigorous mathematical proofs for two-layer networks and single-neuron networks.
result GD trajectories align on a specific bifurcation diagram independent of initialization.
Self-improvement refines language models by verifying their own outputs.
problem Improving language models without external feedback.
method Formalizing self-improvement as sharpening, using the model itself as a verifier.
result RLHF-based self-improvement can outperform SFT-based methods.
Outliers with opposing signals significantly affect neural network optimization.
problem Understanding and mitigating the impact of outliers with opposing signals on neural network training.
method Identifying and analyzing pairs of outliers with strong opposing signals in training data.
result Outliers with opposing signals can cause optimization to enter a narrow valley, leading to oscillatory behavior and eventual loss spikes.
The study analyzes sharpness dynamics in neural networks, revealing mechanisms and conditions.
problem Understanding sharpness in neural network training.
method Fixed point analysis and edge of stability analysis in a simplified 2-layer linear network.
result Reveals mechanisms behind sharpness trends, conditions for edge of stability, and a period-doubling route to chaos.
Weight decay stabilizes training dynamics by slowing progressive sharpening.
problem Understanding how weight decay affects training stability in deep learning models.
method Analyzing weight decay effects at the Edge of Stability, developing a mathematical framework.
result Weight decay dampens oscillations and stabilizes sharpness in CNNs, causing a phase transition in MLPs.
Gradient descent converges with arbitrary stepsize for separable data under Fenchel-Young losses.
problem Understanding the conditions under which gradient descent converges with arbitrary stepsize.
method Using Fenchel-Young losses and leveraging the classical perceptron argument to derive convergence rates.
result GD converges with arbitrary stepsize for a majority of Fenchel-Young losses, with better rates for specific loss functions.
Paper proposes a method to reduce hallucinations in diffusion models using Laplacian score sharpening.
problem Hallucinations in diffusion models create incoherent or unrealistic samples.
method Post-hoc adjustment to the score function during inference using Laplacian approximation.
result Significantly reduces the rate of hallucinated samples across various data types.
We propose a symmetric graph convolutional autoencoder which produces a low-dimensional latent representation from a graph. In contrast to the existing graph autoencoders with asymmetric decoder parts, the proposed autoencoder has a newly designed decoder which builds a completely symmetric autoencoder form. For the re…
Graph convolutions can enhance high frequencies, leading to over-sharpening.
problem Graph convolutions suffer from over-smoothing and poor performance on heterophilic graphs.
method Rigorously prove that linear graph convolutions minimize a generalized Dirichlet energy, showing that weight matrices induce edge-wise attraction or repulsion.
result Graph convolutions can enhance high frequencies, leading to over-sharpening instead of over-smoothing.
Paper tackles noisy labels in deep learning networks.
problem Learning with noisy labels in deep neural networks.
method Sparse regularization strategy to approximate one-hot constraint.
result Improves performance of commonly-used loss functions in noisy labels and class imbalance.
The paper shows how warming up the learning rate improves deep learning performance.
problem Improving deep learning performance through better handling of larger learning rates.
method Systematic experiments with SGD and Adam showing the benefits of warmup and different regimes of operation.
result Properly choosing ηextinit can eliminate the need for warmup and improve performance. Study sharpens threshold for matching correlated graphs without labels.
problem Matching latent vertex correspondences in correlated random graphs.
method Analyzes information-theoretic limits for correct vertex matching in sub-sampled graphs.
result Establishes a sharp information-theoretic threshold for vertex matching recovery.
New theorem shows curvature concentration depends linearly on volume ratio.
problem Gap theorem for nonnegative Ricci curvature manifolds with small curvature concentration.
method Exhibited Ricci flow solution with faster than 1/t curvature decay.
result Curvature concentration depends linearly on asymptotic volume ratio.
This paper addresses Cheeger and Gromoll's question of which vector bundles admit a complete metric of nonnegative curvature, and relates their question to the issue of which sphere bundles admit a metric of positive curvature. We show that any vector bundle which admits a metric of nonnegative curvature must admit a c…
Sharp bounds for high-probability estimation of discrete distributions.
problem Estimating discrete distributions with high probability under χ2-divergence. method Sharp upper and lower bounds for the classical Laplace estimator, and characterization of minimax high-probability risk for any estimator.
result Sharp bounds for high-probability estimation of discrete distributions can be achieved through a simple smoothing strategy.
We introduce a scalable measure of curvature for analyzing training dynamics of large language models.
problem Analyzing the training dynamics of large language models due to high computational cost of measuring Hessian sharpness.
method We introduce critical sharpness and relative critical sharpness as computationally efficient measures capturing Hessian sharpness phenomena.
result We provide the first demonstration of sharpness phenomena at scale up to 7B parameters.
Improved convergence rates for MLE in mixture models using penalized log-likelihood.
problem Convergence rates for MLE in finite mixture models.
method Penalizing log-likelihood to discourage vanishing mixing weights, using Wasserstein distance and new loss functions.
result Improved convergence rates for some mixture components, faster than traditional methods.
The paper sharpens inequalities in hyperbolic spaces.
problem Estimating hyperbolic capacities accurately.
method Detailed theorems establishing sharp capacitary inequalities.
result Established four types of sharp capacitary inequalities.
Detecting edge correlation between two graphs sharpens a threshold based on densest subgraph.
problem Detecting edge correlation between two Erdős-Rényi graphs.
method Formulated as a hypothesis testing problem, connecting to densest subgraph detection.
result Sharp information-theoretic threshold established for edge correlation detection.
We study the Gassner representation of the pure braid group Pn by considering its restriction to a free subgroup F. The kernel of the restriction is shown to lie in the subgroup [Γ3F,Γ2F], sharpening a result of Lipschutz.
The paper explores how symmetries and noise in SGD influence parameter dynamics.
problem Understanding the dynamics of parameter updates in SGD with symmetries.
method Proved the existence of noise equilibria and showed their role in balancing gradient noise.
result Gradient noise creates a systematic motion of parameters to a unique fixed point, called noise equilibria.
New model shows SGD can prefer sharp or flat solutions based on label noise.
problem Understanding SGD's preference for flat or sharp solutions during training.
method Solved an analytically solvable model to explore SGD behavior.
result Data distribution determines sharpness at convergence; isotropic label noise leads to flat minimum preference.
We give tight concentration bounds for mixtures of martingales that are simultaneously uniform over (a) mixture distributions, in a PAC-Bayes sense; and (b) all finite times. These bounds are proved in terms of the martingale variance, extending classical Bernstein inequalities, and sharpening and simplifying prior wor…
Differential privacy of Gaussian process posterior sampling
problem Privacy of posterior sample paths from Gaussian process
method Intrinsic randomness yields DP guarantees
result Intrinsic randomness yields DP guarantees
We sharpen the construction of representation space in the paper "Principal Series Representations of Infinite Dimensional Lie Groups II: Construction of Induced Representations". We show that the principal series representation spaces constructed there, are completions of spaces of sections of Hilbert bundles rather t…
Hamiltonian Monte Carlo (HMC) is a state-of-the-art Markov chain Monte Carlo sampling algorithm for drawing samples from smooth probability densities over continuous spaces. We study the variant most widely used in practice, Metropolized HMC with the Störmer-Verlet or leapfrog integrator, and make two primary contribut…
We give a new lower bound for the first gap λ2−λ1 of the Dirichlet eigenvalues of the Schr{ö}dinger operator on a bounded convex domain Ω in Rn or Sn and greatly sharpens the previous estimates. The new bound is explicit and computable.
This paper sharpens privacy guarantees for high-dimensional PCA under differential privacy.
problem Understanding the exact privacy loss in high-dimensional PCA with differential privacy.
method Analyzes the exponential mechanism in a model-free setting for high-dimensional PCA.
result Sharp utility and privacy characterizations in high dimensions show the difficulty of detecting a target individual's presence.
In this work, complete constant mean curvature 1 (CMC-1) surfaces in hyperbolic 3-space with total absolute curvature at most 4 pi are classified. This classification suggests that the Cohn-Vossen inequality can be sharpened for surfaces with odd numbers of ends, and a proof of this is given.
Gradient descent at edge of stability stabilizes implicitly, following projected gradient descent.
problem Gradient descent's stability and sharpness behavior at the edge of instability.
method Cubic Taylor expansion analysis of gradient descent dynamics.
result Gradient descent at edge of stability implicitly follows projected gradient descent.
We further sharpen higher type adjunction inequalities of P. Ozsváth and Z. Szabó on a 4-manifold M with a nonzero Seiberg-Witten invariant for a Spinc structure s, when an embedded surface Σ⊂M satisfies [Σ]⋅[Σ]≥0 and ∣⟨[Σ],c1(s)⟩∣+[Σ]⋅[Σ]≥2b1(M).
The goal of the paper is to sharpen and generalise bounds involving the Cheeger's isoperimetric constant h and the first eigenvalue λ1 of the Laplacian. A celebrated lower bound of λ1 in terms of h, λ1≥h2/4, was proved by Cheeger in 1970 for smooth Riemannian manifolds. An upper bound on $λ_{1…
New method for sequential probability assignment reduces regret using contextual Shtarkov sums.
problem Minimizing regret in sequential probability assignment with arbitrary hypothesis classes.
method Introducing contextual Shtarkov sum and contextual Normalized Maximum Likelihood (cNML) algorithm.
result The contextual Shtarkov sum characterizes minimax regret and provides a minimax optimal strategy.
Let X be a locally symmetric space defined by a simple Chevalley group G and a congruence subgroup of G(Q). In this generality, the Weyl law for X was proved by Lindenstrauss--Venkatesh. In the case where G is simply connected, we sharpen their result by giving a power saving estimate for the remainde…
Transformers approximate Bayesian posteriors but not exactly.
problem Bayesian accounts of in-context learning face challenges due to task-preserving order changes in transformers.
method Showed that excess prequential code length is exactly cumulative predictive KL, decomposing expected regret into order-averaged predictor and order-averaging gain.
result Transformers approximate Bayesian posteriors but not exactly, priced by log loss.
From concentration inequalities for the suprema of Gaussian or Rademacher processes an inequality is derived. It is applied to sharpen existing and to derive novel bounds on the empirical Rademacher complexities of unit balls in various norms appearing in the context of structured sparsity and multitask dictionary lear…
The study sharpens local Bernstein estimates for Laplace eigenfunctions on compact manifolds.
problem Understanding local growth properties of Laplace eigenfunctions on compact Riemannian manifolds.
method Refined Donnelly-Fefferman method based on L2--Carleman estimates, combined with elliptic regularity and patching of local Carleman estimates. result Almost sharp local Lp--Bernstein inequalities for p∈[1,∞]. Hyperfitting improves LLM generation quality by enhancing diversity, contrary to simple temperature scaling.
problem Improving open-ended generation quality of LLMs with minimal fine-tuning effort.
method Demonstrates that hyperfitting, a phenomenon where LLMs are fine-tuned to near-zero training loss, enhances generation quality and mitigates repetition.
result Hyperfitting is distinct from temperature scaling and involves a dynamic, context-dependent rank reordering mechanism in the final transformer block.
Colding and Minicozzi have shown that an embedded minimal disk 0∈Σ⊂BR in $\Real^3$ with large curvature at 0 looks like a helicoid on the scale of R. Near 0, this can be sharpened: on the scale of ∣A∣−1(0), Σ is close, in a Lipschitz sense, to a piece of a helicoid. We use surfaces constructed by C…
Quandle cocycle invariants form a powerful and well developed tool in knot theory. This paper treats their variations - namely, positive and twisted quandle cocycle invariants, and shadow invariants. We interpret the former as particular cases of the latter. As an application, several constructions from the shadow worl…
NM-PPG optimizes adaptive feature acquisition in POMDPs for better predictions.
problem Optimizing adaptive feature acquisition in prediction problems with costly features.
method Non-myopic pathwise policy gradients (NM-PPG) with continuous relaxation and straight-through rollout.
result NM-PPG outperforms state-of-the-art AFA methods on synthetic and real-world datasets.
Given a data matrix X∈Rn×d and a response vector y∈Rn, suppose n>d, it costs O(nd2) time and O(nd) space to solve the least squares regression (LSR) problem. When n and d are both large, exactly solving the LSR problem is very expensive. When n≫d, one feasible approach to spee…
In this paper we first give a one-move version of Markov's braid theorem for knot isotopy in S3 that sharpens the classical theorem. Then a relative version of Markov's theorem concerning a fixed braided portion in the knot. We also prove an analogue of Markov's theorem for knot isotopy in knot complements. Finally …
Bayesian Attention Networks compress data by focusing on key training samples.
problem Lossless data compression for efficiency.
method Bayesian Attention Networks with attention factors and latent space.
result Efficient prediction using a few correlated training samples.