SGD's performance improves with critical batch size, minimizing SFO complexity.
problem Optimizing SGD's performance with batch size and learning rate.
method Analysis of SGD using constant and decaying learning rates, focusing on batch size effects.
result SGD with critical batch size minimizes SFO complexity.
DPSM minimizes prediction set size by integrating conformal principles into deep classifier training.
problem Large prediction sets from standard conformal methods are impractical.
method Formulates conformal training as bilevel optimization, proposing DPSM algorithm.
result Significantly reduces prediction set size compared to prior methods.
We speed up marginal inference by ignoring factors that do not significantly contribute to overall accuracy. In order to pick a suitable subset of factors to ignore, we propose three schemes: minimizing the number of model factors under a bound on the KL divergence between pruned and full models; minimizing the KL dive…
Loss minimization leads to multicalibration for neural networks.
problem Ensuring fairness in predictions across multiple protected groups.
method Minimizing squared loss over neural networks of size n.
result Minimizing loss over neural nets of size n implies multicalibration for most values of n.
We study the Stochastic Gradient Descent (SGD) method in nonconvex optimization problems from the point of view of approximating diffusion processes. We prove rigorously that the diffusion process can approximate the SGD algorithm weakly using the weak form of master equation for probability evolution. In the small ste…
Optimal batch size minimizes training time for neural networks.
problem Minimizing training time for two-layer neural networks with SGD.
method Characterized optimal batch size as a function of target hardness (information exponents). Used Correlation loss SGD to overcome limitations.
result Optimal batch size minimizes training time without changing total sample complexity.
This paper classifies chiral graphs up to size 12.
problem Understanding the chirality of simple graphs to predict molecular behavior.
method Classifying minor minimal intrinsically chiral graphs among simple graphs of size up to 12.
result Complete set of minor minimal graphs for intrinsic properties of chiral molecules.
ResNets minimize circuit size for fitting data in HTMC regime.
problem Finding the simplest algorithm that fits data.
method Defining HTMC and ResNet norms to relate circuit size and function fitting.
result Minimizing ResNet norm is equivalent to finding a circuit with minimal nodes.
This a survey on a series of recent papers in collaboration with Emanuele Spadaro on the regularity of area-minimizing currents in codimension higher than 1.
Permutation of any two hidden units yields invariant properties in typical deep generative neural networks. This permutation symmetry plays an important role in understanding the computation performance of a broad class of neural networks with two or more hidden units. However, a theoretical study of the permutation sy…
New algorithms verify and search causal graphs with minimal interventions.
problem Recovering causal graphs from interventional data.
method Characterization of minimal intervention sets for verification, and adaptive graph separator algorithm for search.
result First provable algorithms for efficient verification and search of causal graphs.
Minimal submanifolds in matrix spaces proven for specific ranks.
problem Minimal submanifolds in matrix spaces.
method Proving semialgebraic sets of matrices are minimal.
result Rectangular, skew-symmetric, and symmetric matrices with prescribed eigenvalues are minimal.
This is the third paper in a series devoted to enumerating the prime alternating knots and links. This paper establishes a method for enumerating the prime alternating links. It is shown that one may choose any prime alternating link diagram of a given minimal crossing size and by applications of just two operators (T …
We prove that the rank (that is, the minimal size of a generating set) of lattices in a general connected Lie group is bounded by the co-volume of the projection of the lattice to the semi-simple part of the group. This was proved by Gelander for semi-simple Lie groups and by Mostow for solvable Lie groups. Here we con…
Let Fg be a closed orientable surface of genus g. A set Ω={γ1,…,γs} of pairwise non-homotopic simple closed curves on Fg is called a \emph{filling system} or simply a \emph{filling} of Fg, if Fg∖Ω is a union of b topological discs for some b≥1. A filling system is called \em…
We show the rank (i.e. minimal size of a generating set) of lattices cannot grow faster than the volume.
New adaptive scheduler improves SAM for better model training.
problem Training machine learning models requires selecting a learning rate, which is often difficult and time-consuming.
method Derive Polyak schedulers tailored to SAM-style updates, proving linear convergence for strongly convex objectives and an O(1/T) rate for convex objectives.
result Polyak schedulers achieve comparable or better performance than tuned SAM baselines, reducing the need for learning-rate tuning.
New study shows ERMs can fail in convex optimization with high dimensionality.
problem The limitations of Empirical Risk Minimizer in high-dimensional stochastic convex optimization.
method Constructed a specific instance showing ERMs can be unique and overfit.
result Gradient Descent can also overfit in certain conditions, resolving a gap in lower bounds.
Classifies knots by lattice size, finding unknot ratios and crossing numbers.
problem Understanding the distribution of knots within different lattice sizes.
method Introduced a new knot classification by lattice size, analyzed ratios of unknots and knots with more than 10 crossings, and compared with theoretical estimates.
result Ratio of unknots decreases exponentially with lattice size, and computational results match theoretical estimates.
Classifies area-minimizing surfaces in R^4 as algebraic.
problem Classifying entire area-minimizing surfaces in R^4.
method Using quadratic area growth and holomorphic polynomials to cut out surfaces.
result Entire 2-dimensional area-minimizing or stable surfaces in R^4 are algebraic.
We consider large scale empirical risk minimization (ERM) problems, where both the problem dimension and variable size is large. In these cases, most second order methods are infeasible due to the high cost in both computing the Hessian over all samples and computing its inverse in high dimensions. In this paper, we pr…
New mathematical framework proves the effectiveness of reducing neural network sizes.
problem Selecting optimal neural network sizes to avoid overfitting.
method Adaptive group Lasso applied to one-hidden-layer feedforward networks.
result Adaptive group Lasso is consistent and can accurately reconstruct network sizes.
RSGDA improves convergence rates for nonconvex-strongly concave optimization.
problem Optimization of nonconvex-strongly concave problems.
method Randomized Stochastic Gradient Descent Ascent (RSGDA) with optimal loop sizes.
result First almost sure convergence rates for SGDA algorithms on nonconvex-strongly concave settings.
We consider the problem of exact recovery of any m×n matrix of rank ϱ from a small number of observed entries via the standard nuclear norm minimization framework. Such low-rank matrices have degrees of freedom (m+n)ϱ−ϱ2. We show that any arbitrary low-rank matrices can be recovered exa…
Improved Frank-Wolfe method reduces dependence on data size for empirical risk minimization.
problem Reducing dependence on number of data observations in Frank-Wolfe methods.
method Taylor-series approximated gradients applied to Frank-Wolfe method.
result Significant speed-ups over existing methods on real-world datasets.
Scaling laws for neural language models reveal optimal model size and compute allocation.
problem Understanding the optimal model size and compute allocation for neural language models.
method Empirical analysis of scaling laws for cross-entropy loss across model size, dataset size, and compute.
result Simple equations govern the dependence of overfitting and training speed on model/dataset size and model size, respectively.
We prove the two theorems of the title, settling two long standing questions in the local theory of singular minimal hypersurfaces. The sharpness of either result is with respect to its hypothesis on the size of the allowable singular sets. The proofs of both theorems rely heavily on the author's recent regularity and …
We extend the results of our recent preprint [arXiv: 1811.00515] into higher dimensions n≥4. For minimizing harmonic maps u∈W1,2(Ω,S2) from n-dimensional domains into the two dimensional sphere we prove: (1) An extension of Almgren and Lieb's linear law, namely \[\mathcal{H}^{n-3}(\textrm{sin…
Adaptive SGD learns optimal batch size for strong convex functions.
problem Finding optimal batch size for SGD in practice.
method Adaptive SGD method that learns optimal batch size.
result Adaptive SGD exhibits nearly optimal performance in experiments.
Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the …
Kernel adaptive filters (KAF) are a class of powerful nonlinear filters developed in Reproducing Kernel Hilbert Space (RKHS). The Gaussian kernel is usually the default kernel in KAF algorithms, but selecting the proper kernel size (bandwidth) is still an open important issue especially for learning with small sample s…
The braid group's commutator subgroup is generated by two elements for n ≥ 7.
problem Generating the smallest possible generating sets for the commutator subgroup of braid groups.
method Analyzing specific cases of braid groups (n=4, 6, 5, 7+) to find generating sets of minimal size.
result For n ≥ 7, the commutator subgroup of the braid group is generated by two elements.
Financial undertakings often have to deal with liabilities of the form 'non-hedgeable claim size times value of a tradeable asset', e.g. foreign property insurance claims times fx rates. Which strategy to invest in the tradeable asset is risk minimal? We generalize the Gram-Charlier series for the sum of two dependent …
Improved estimates for singularities in capillary surfaces.
problem Understanding the singularities of minimizing capillary hypersurfaces.
method Improved estimates based on connections to the one-phase Bernoulli problem.
result The singular set is of codimension at least 4, improving for specific angles.
New method reduces training time for deep hedging networks.
problem Challenges in training deep hedging networks with large batch sizes.
method Integrates topological features to reduce batch sizes.
result Practical training of deep hedging models without sacrificing performance.
IMPACT optimizes LLM compression by focusing on activation importance, reducing model size up to 55.4%.
problem Resource constraints in deploying large language models (LLMs).
method IMPACT integrates activation importance into low-rank compression, optimizing for both size and accuracy.
result IMPACT achieves up to 55.4% greater model size reduction while maintaining comparable or better accuracy.
Research shows minimal communication limits adaptive function estimation rates.
problem Adaptive estimation of a smooth function under minimal communication constraints.
method Investigates the L∞-risk and L2-risk under different numbers of servers. result For L∞-risk, optimal rates cannot be achieved under minimal communication. For L2-risk, adaptivity is possible but depends on server number and sample size. Discrete approximation solves Björling's minimal surface problem.
problem Constructing minimal surfaces from real-analytic curves with specified normal fields.
method Approximate solution by discrete minimal surfaces and discrete isothermic surfaces.
result Approximation error is proportional to the square of the mesh size.
We revisit resampling procedures for error estimation in binary classification in terms of U-statistics. In particular, we exploit the fact that the error rate estimator involving all learning-testing splits is a U-statistic. Thus, it has minimal variance among all unbiased estimators and is asymptotically normally dis…
New method combines experimental and observational data for causal inference.
problem Combining internal validity of experiments and larger sample sizes of observations.
method Empirical risk minimization (ERM) framework with cross-validation.
result Efficacy and reliability demonstrated on real and synthetic data.
The variance reduction class of algorithms including the representative ones, SVRG and SARAH, have well documented merits for empirical risk minimization problems. However, they require grid search to tune parameters (step size and the number of iterations per inner loop) for optimal performance. This work introduces `…
Optimal microlending group size is 5 people.
problem Determining the best group size for microlending to minimize default risk.
method Mathematical modeling with interacting forces and precise hypotheses.
result The optimal microlending group size is 5 people.
Binary Stochastic Filtering (BSF), the algorithm for feature selection and neuron pruning is proposed in this work. The method defines filtering layer which penalizes amount of the information involved in the training process. This information could be the input data or output of the previous layer, which directly lead…
Adaptive step-size improves optimization in complex geometries.
problem Optimizing functions with non-Euclidean geometries.
method Adaptive step-size strategy for optimization algorithms.
result Guaranteed convergence for Adaptive Conditional Gradient Descent.
Paper proposes a cost-sensitive conformal training method with provably controllable learning bounds.
problem Uncertainty quantification and learning bounds in conformal prediction.
method Cost-sensitive conformal training algorithm that minimizes the expected size of prediction sets using rank weighting.
result Theoretical analysis shows tightness between weighted objective and expected size of conformal prediction sets.
Gradient descent with large steps leads to chaotic parameter space and unpredictable outcomes.
problem Understanding the behavior of gradient descent with large step sizes in matrix factorization.
method Analyzing the fractal structure of the parameter space and deriving critical step sizes for convergence.
result Gradient descent with large steps exhibits chaotic behavior and sensitivity to initialization, creating a fractal boundary between converging and diverging minimizers.
Minimal surfaces in 4D space are stable if their Gauss map spherical area is less than 2π.
problem Stability of minimal surfaces in 4D space.
method Geometric criteria based on the Gauss map of minimal surfaces in terms of the spherical area.
result Minimal surfaces in 4D space are stable if their Gauss map spherical area is less than 2π.
Paper finds optimal mini-batch size for SGD to speed up learning.
problem Optimizing mini-batch size for faster SGD convergence.
method Empirical inverse law and theoretical bound on mini-batch SGD training.
result An accurate model for predicting training time and identifying implications for algorithm and hardware.