arXiv research

Proposes a more robust rating scale for banks.

Unintended effects from scaling neural network outputs with adaptive learning rates.

Exploiting low-rank structure of the user-item rating matrix has been the crux of many recommendation engines. However, existing recommendation engines force raters with heterogeneous behavior profiles to map their intrinsic rating scales to a common rating scale (e.g. 1-5). This non-linear transformation of the rating…

Most optimizers including stochastic gradient descent (SGD) and its adaptive gradient derivatives face the same problem where an effective learning rate during the training is vastly different. A learning rate scheduling, mostly tuned by hand, is usually employed in practice. In this paper, we propose CProp, a gradient…

Quantifying the degree of atrophy is done clinically by neuroradiologists following established visual rating scales. For these assessments to be reliable the rater requires substantial training and experience, and even then the rating agreement between two radiologists is not perfect. We have developed a model we call…

We introduce a solvable model of randomly growing systems consisting of many independent subunits. Scaling relations and growth rate distributions in the limit of infinite subunits are analysed theoretically. Various types of scaling properties and distributions reported for growth rates of complex systems in a variety…

We introduce a new weight-decay scaling rule to maintain sublayer gains across different widths in modern scale-invariant architectures.

Large learning rates work surprisingly well in standard parameterization, contrary to theory.

Classical scaling is shown to be optimal under various noisy conditions.

The paper introduces gapped scale-sensitive dimensions to improve learning rate bounds.

In this paper we investigate the scaling behavior of the average daily exchange rate returns of the Indian Rupee against four foreign currencies namely US Dollar, Euro, Great Britain Pound and Japanese Yen. Average daily exchange rate return of the Indian Rupee against US Dollar is found to exhibit a persistent scaling…

We introduce a simple agent-based model which allows us to analyze three stylized facts: a fat-tailed size distribution of companies, a `tent-shaped' growth rate distribution, the scaling relation of the growth rate variance with firm size, and the causality between them. This is achieved under the simple hypothesis th…

For the purpose of elucidating the correlation among currencies, we analyze daily and high-resolution data of foreign exchange rates. There is strong correlation for pairs of currencies of geographically near countries. We show that there is a time delay of order less than a minute between two currency markets having a…

Paper analyzes convergence rates of two time-scale AC and NAC algorithms.

Paper studies MCCR models with scale parameters tending to zero, revealing optimal learning rate and comparing robustness.

We discover scaling laws for kernel regression loss under various learning rate schedules.

Unified learning-rate scale for CNNs and ResNets, avoiding depth imbalance.

This research examines how the error rate of nearest neighbor classifiers varies with dataset size.

This paper quantifies hyperparameter transfer and finds embedding layer learning rate is key.

Develops methods to measure and set function-space learning rates in neural networks.

Study no-arbitrage conditions in 1D diffusion markets with interest rates.

New learning rates derived for Tikhonov-regularized problems without kernel assumptions.

A universal learner achieves best rates for all distributions.

We investigate finite-time decoupled convergence in nonlinear two-time-scale stochastic approximation.

In theoretical analysis of deep learning, discovering which features of deep learning lead to good performance is an important task. In this paper, using the framework for analyzing the generalization error developed in Suzuki (2018), we derive a fast learning rate for deep neural networks with more general activation …

Study rates of convergence for approximate solutions to linear ill-posed problems in Hilbert scales.

Optimal scaling found to depend on operator norm across large models and datasets.

New learning dynamics achieve fast convergence in games without needing to know utility scales.

We study two time-scale linear stochastic approximation algorithms, which can be used to model well-known reinforcement learning algorithms such as GTD, GTD2, and TDC. We present finite-time performance bounds for the case where the learning rate is fixed. The key idea in obtaining these bounds is to use a Lyapunov fun…

Proposes GBBHE for efficient large-scale regression.

New research shows unlabeled data is equally valuable as labeled data in certain semi-supervised learning scenarios.

In this paper, we consider daily financial data of a collection of different stock market indices, exchange rates, and interest rates, and we analyze their multi-scaling properties by estimating a simple specification of the Markov-switching multifractal model (MSM). In order to see how well the estimated models captur…

ScheduleFree+ improves large language model training without schedules or learning rates.

Optimal learning rates decay to zero in easy tasks and maintain a warmup phase in hard tasks.

Study tests rough fractional volatility model across different time scales, revealing new volatility patterns.

Training deep neural networks using a large batch size has shown promising results and benefits many real-world applications. However, the optimizer converges slowly at early epochs and there is a gap between large-batch deep learning optimization heuristics and theoretical underpinnings. In this paper, we propose a no…

We analyze the probability of ruin for the {\it scaled} classical Cramér-Lundberg (CL) risk process and the corresponding diffusion approximation. The scaling, introduced by Iglehart \cite{I1969} to the actuarial literature, amounts to multiplying the Poisson rate $\la$ by $n$, dividing the claim severity by $\sqrtn$, …

Efficient knockoffs for large-scale feature selection.

We construct a general stochastic process and prove weak convergence results. It is scaled in space and through the parameters of its distribution. We show that our simplified scaling is equivalent to time scaling used frequently. The process is constructed as an integral with respect to a Poisson random measure which …

Optimization of neural networks scales with γ, revealing unique loss curves and optimal learning rates.

Algorithm improves online canonical correlation analysis.

AdaScale SGD adapts learning rates for large-batch training efficiently.

We investigate how the final parameters found by stochastic gradient descent are influenced by over-parameterization. We generate families of models by increasing the number of channels in a base network, and then perform a large hyper-parameter search to study how the test error depends on learning rate, batch size, a…

New insights into learning rates and batch sizes for neural networks using random matrix theory.

Sharp theory of neural network scaling laws for hierarchical targets.

We establish decoupled functional CLTs for two-time-scale stochastic approximation.

Study uniform rates for estimating Gaussian mixtures without separation assumption.

The miltifractal properties and scaling behaviour of the exchange rate variations of the Iranian rial against the US dollar from a daily perspective is numerically investigated. For this purpose the multifractal detrended fluctuation analysis (MF-DFA) is used. Through multifractal analysis, the scaling exponents, gener…