arXiv research

We investigate finite-time decoupled convergence in nonlinear two-time-scale stochastic approximation.

Study reveals how initialization scale affects training accuracy in linear networks.

Scaling laws in linear regression explain model performance improvements with size and data.

Wide neural networks become linear, with constant tangent kernel, due to Hessian scaling.

Improved scaling laws in linear regression using data reuse.

This work studies scaling laws for low-precision training in high-dimensional linear regression.

Improved bounds for non-linear SA with fast convergence.

The price of financial assets are, since Bachelier, considered to be described by a (discrete or continuous) time sequence of random variables, i.e a stochastic process. Sharp scaling exponents or unifractal behavior of such processes has been reported in several works. In this letter we investigate the question of sca…

The detrending moving average (DMA) algorithm is one of the best performing methods to quantify the long-term correlations in nonstationary time series. Many long-term correlated time series in real systems contain various trends. We investigate the effects of polynomial trends on the scaling behaviors and the performa…

New bounds on self-normalized martingales improve online linear regression performance.

Simplifies deep learning scaling analysis without sacrificing accuracy.

SignSGD outperforms SGD in linear regression with optimal scaling laws under PLRF model.

This work bridges two views of feature learning in neural networks.

We study scale invariant but not necessarily conformal invariant deformations of non-relativistic conformal field theories from the dual gravity viewpoint. We present the corresponding metric that solves the Einstein equation coupled with a massive vector field. We find that, within the class of metric we study, when w…

Study rates of convergence for approximate solutions to linear ill-posed problems in Hilbert scales.

A new method resolves permutation issues in shuffled linear regression for large-scale applications.

We present an idea of unifying small scale (topology, proximity spaces, uniform spaces) and large scale (coarse spaces, large scale spaces). It relies on an analog of multilinear forms from Linear Algebra. Each form has a large scale compactification and those include all well-known compactifications: Higson corona, Gr…

Linear memory stores associations up to a logarithmic scale, but listwise retrieval can handle a quadratic scale.

A topology on a set $X$ is the same as a projection (i.e. an idempotent linear operator) $cl:2^X\to 2^X$ satisfying $A\subset cl(A)$ for all $A\subset X$. That's a good way to summarize Kuratowski's closure operator. Basic geometry on a set $X$ is a dot product $\cdot:2^X\times 2^X\to 2^Y$. Its equivalent form is an or…

This paper improves linear system solving by optimizing matrix diagonal scaling.

Efficient knockoffs for large-scale feature selection.

Scaled sparse linear regression jointly estimates the regression coefficients and noise level in a linear model. It chooses an equilibrium with a sparse regression method by iteratively estimating the noise level via the mean residual square and scaling the penalty in proportion to the estimated noise level. The iterat…

SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.

Proposes a method for differentially private linear regression and synthetic data generation.

Unified analysis of parameter norms in overparameterized linear models, revealing scaling laws and thresholds.

JULIA combines multi-linear and nonlinear models for tensor completion.

The paper analyzes how repeating epochs affects data scaling in linear regression.

SCAFFLSA reduces communication complexity for federated learning with heterogeneous clients.

Theory of ends of spaces using linear algebra.

This work extends the scaling law to multiple and kernel regression, challenging traditional machine learning principles.

Temperature scaling improves model uncertainty but not diversity in LLMs.

Recently, self-normalizing neural networks (SNNs) have been proposed with the intention to avoid batch or weight normalization. The key step in SNNs is to properly scale the exponential linear unit (referred to as SELU) to inherently incorporate normalization based on central limit theory. SELU is a monotonically incre…

Preprocessing data is an important step before any data analysis. In this paper, we focus on one particular aspect, namely scaling or normalization. We analyze various scaling methods in common use and study their effects on different statistical learning models. We will propose a new two-stage scaling method. First, w…

The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention lookup scale linearly in the size of the attended sequence. Second, it does not enc…

Combining neural networks and multiscale decomposition for financial market analysis.

We present a new method for high-dimensional linear regression when a scale parameter of the additive errors is unknown. The proposed estimator is based on a penalized Huber $M$-estimator, for which theoretical results on estimation error have recently been proposed in high-dimensional statistics literature. However, t…

New method reduces ensemble size for linear bandits, achieving near optimal regret.

Natural image statistics exhibit hierarchical dependencies across multiple scales. Representing such prior knowledge in non-factorial latent tree models can boost performance of image denoising, inpainting, deconvolution or reconstruction substantially, beyond standard factorial "sparse" methodology. We derive a large …

We consider online learning with linear models, where the algorithm predicts on sequentially revealed instances (feature vectors), and is compared against the best linear function (comparator) in hindsight. Popular algorithms in this framework, such as Online Gradient Descent (OGD), have parameters (learning rates), wh…

New method speeds up Gaussian process training and inference for large datasets.

New CAGP-GS framework improves Gaussian process uncertainty quantification.

Optimal multiscale learning of linear operators

We extend Kirman's model by introducing variable event time scale. The proposed flexible time scale is equivalent to the variable trading activity observed in financial markets. Stochastic version of the extended Kirman's agent based model is compared to the non-linear stochastic models of long-range memory in financia…

We analyzed optimism in linear and kernel regression models.

Geometrically transforms nonconservative dynamics to linearize Kepler and Manev systems.

Learning to control linear systems is statistically hard, especially for underactuated systems.

Study on neural scaling laws for solving linear systems in-context.

Efficient algorithms speed up adversarial training for linear models.