1-D CNNs classify pupil size variations in scotopic conditions.
problem Handling wide inter-subjects variability in pupil analysis.
method 1-D Convolutional Neural Networks applied directly to raw pupil size data.
result 1-D CNNs provide high accuracy in classifying short-range pupil size sequences.
We first, introduce a deep learning based framework named as DeepIrisNet2 for visible spectrum and NIR Iris representation. The framework can work without classical iris normalization step or very accurate iris segmentation; allowing to work under non-ideal situation. The framework contains spatial transformer layers t…
We present a large scale data set, OpenEDS: Open Eye Dataset, of eye-images captured using a virtual-reality (VR) head mounted display mounted with two synchronized eyefacing cameras at a frame rate of 200 Hz under controlled illumination. This dataset is compiled from video capture of the eye-region collected from 152…
In our previous study, we introduced stable specification search for cross-sectional data (S3C). It is an exploratory causal method that combines stability selection concept and multi-objective optimization to search for stable and parsimonious causal structures across the entire range of model complexities. In this st…
Recent advances in deep learning have facilitated the demand of neural models for real applications. In practice, these applications often need to be deployed with limited resources while keeping high accuracy. This paper touches the core of neural models in NLP, word embeddings, and presents a new embedding distillati…
Privacy-preserving eye tracking framework using synthetic images.
problem Preserving sensitive personal information in eye tracking technology.
method Randomized encoding for privacy, Support Vector Regression model training on synthetic eye images.
result Framework achieves real-time gaze estimation with same accuracy as non-private version.
Unlike machines, humans learn through rapid, abstract model-building. The role of a teacher is not simply to hammer home right or wrong answers, but rather to provide intuitive comments, comparisons, and explanations to a pupil. This is what the Learning Under Privileged Information (LUPI) paradigm endeavors to model b…
The ability to accurately predict the fit of fashion items and recommend the correct size is key to reducing merchandise returns in e-commerce. A critical prerequisite of fit prediction is size normalization, the mapping of product sizes across brands to a common space in which sizes can be compared. At present, size n…
A new scaling law predicts optimal batch size for training models.
problem Finding the optimal batch size for training models efficiently.
method Proposed a three-term scaling law that considers model size, training data, training steps, and batch size.
result The three-term law accurately recovers the optimal batch size and can be robustly fit with fewer training runs.
Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a n…
Implicit Q-learning and SARSA adjust step-sizes automatically, improving stability and performance.
problem Numerical instability and slow progress in Q-learning and SARSA due to step-size calibration.
method Reformulate iterative updates as fixed-point equations, scaling step-sizes inversely with feature norms.
result Implicit methods maintain stability over broader step-size ranges and achieve comparable convergence rates.
Riemannian stochastic gradient descent converges faster with increasing batch size.
problem Improving convergence rate of Riemannian stochastic gradient descent.
method Theoretical analysis and numerical investigation of increasing batch size effects.
result Riemannian stochastic gradient descent converges faster with increasing batch size.
Study on size and depth of neural networks for approximating benign functions, showing barriers and explicit results.
problem Understanding how size and depth of neural networks affect their ability to approximate benign functions.
method Analyzing ReLU networks for benign functions, proving barriers and explicit results.
result Explicit benign functions that cannot be approximated by networks of certain sizes or depths, showing barriers to size and depth separation.
When using active learning, smaller batch sizes are typically more efficient from a learning efficiency perspective. However, in practice due to speed and human annotator considerations, the use of larger batch sizes is necessary. While past work has shown that larger batch sizes decrease learning efficiency from a lea…
SGD's performance improves with critical batch size, minimizing SFO complexity.
problem Optimizing SGD's performance with batch size and learning rate.
method Analysis of SGD using constant and decaying learning rates, focusing on batch size effects.
result SGD with critical batch size minimizes SFO complexity.
Convolutional neural networks (CNNs) are commonly trained using a fixed spatial image size predetermined for a given model. Although trained on images of aspecific size, it is well established that CNNs can be used to evaluate a wide range of image sizes at test time, by adjusting the size of intermediate feature maps.…
A tick size is the smallest increment of a security price. It is clear that at the shortest time scale on which individual orders are placed the tick size has a major role which affects where limit orders can be placed, the bid-ask spread, etc. This is the realm of market microstructure and there is a vast literature o…
Adaptive batch size schedules improve language model training efficiency and generalization.
problem Dilemma of choosing batch sizes in large-scale model training.
method General-purpose adaptive batch size schedules compatible with data and model parallelism.
result Adaptive batch size schedules outperform constant batch sizes and heuristic warmup schedules.
A new method SEBS optimizes SGD batch size for better performance.
problem Optimizing batch size for SGD to balance training speed and generalization.
method SEBS method uses a multi-stage geometric batch size enlargement scheme.
result SEBS reduces parameter updates without increasing generalization error.
The sizes of Markov equivalence classes of directed acyclic graphs play important roles in measuring the uncertainty and complexity in causal learning. A Markov equivalence class can be represented by an essential graph and its undirected subgraphs determine the size of the class. In this paper, we develop a method to …
New convergence results for NGVI with various step sizes and sample sizes.
problem Understanding convergence of stochastic NGVI for various schedules.
method Projected stochastic NGVI for exponential family variational distributions.
result Geometric convergence and $\mathcal{O}\left(\frac{1}{T^ρ}
ight)$ rates for different schedules.
Large batch sizes reduce gradient variance in DP-SGD, improving privacy.
problem Understanding why large batch sizes work in DP-SGD.
method Decomposed total gradient variance into subsampling and noise-induced variances, proving batch size independence in the limit.
result Large batch sizes reduce effective total gradient variance, improving privacy in DP-SGD.
Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch s…
We make policy optimization algorithms batch size-invariant by decoupling proximal and behavior policies.
problem Some policy optimization algorithms do not have batch size-invariance, leading to inefficiencies.
method We decouple the proximal policy from the behavior policy to achieve batch size-invariance.
result Our approach makes policy optimization algorithms more efficient and allows them to use stale data more effectively.
The paper analyzes fixed step-size SA schemes on Riemannian manifolds.
problem Developing efficient algorithms for optimization on curved spaces.
method Fixed step-size stochastic approximation schemes in a Riemannian framework.
result The schemes converge to the solution as the step-size approaches zero.
A simple block configures optimal kernel sizes for time series classification.
problem Choosing the right kernel size for time series classification.
method Proposes Omni-Scale block (OS-block) with kernel sizes determined by prime numbers.
result Models with OS-block achieve state-of-the-art performance on time series benchmarks.
Revisits granular models explaining firm growth rates and sizes.
problem Understanding the relationship between firm size and growth rate statistics.
method Developed new theoretical insights linking firm size and growth rate statistics within granular models.
result Growth volatility distribution is size-independent but fat-tailed, challenging granular models.
Study finds optimal vocabulary size for neural machine translation.
problem Imbalanced class distribution in language data affects NMT performance.
method Casts NMT as a classification task, analyzes vocabulary sizes, and tests multiple languages.
result Certain vocabulary sizes outperform others, explaining NMT performance.
We determine the critical batch size for large language models and find it scales with data size, not model size.
problem Determining the optimal batch size for large-scale model training.
method We propose a measure of critical batch size, pre-trained models, and systematic hyper-parameter sweeps.
result The critical batch size scales primarily with data size, not model size.
Polyak step size GD reaches final radius of convergence after log iterations.
problem Statistical and computational complexities of Polyak step size GD.
method Generalized smoothness and Lojasiewicz conditions, stability of gradients.
result Polyak step size GD reaches final statistical radius of convergence after logarithmic number of iterations.
The paper analyzes the expected size of conformal prediction sets.
problem Lack of finite-sample analysis and guarantees for prediction set sizes.
method Theoretical quantification and empirical computation of expected set size.
result Derives point estimates and high-probability interval bounds for prediction set size.
The paper analyzes and validates two step size schedules for SGD: exponential and cosine, proving their adaptivity and performance.
problem The variability of SGD performance due to step size choice.
method Analysis and empirical evaluation of exponential and cosine step sizes.
result Exponential and cosine step sizes are adaptive to noise and achieve optimal performance without tuning hyperparameters.
Adaptive batch sizes improve local gradient methods in distributed training.
problem Communication bottlenecks in distributed deep learning.
method Adaptive batch size strategies for local gradient methods.
result Adaptive batch sizes reduce minibatch gradient variance and improve training efficiency.
Using detailed statistical analyses of the size distribution of a universe of equity exchange-traded funds (ETFs), we discover a discrete hierarchy of sizes, which imprints a log-periodic structure on the probability distribution of ETF sizes that dominates the details of the asymptotic tail. This allows us to propose …
AutoStep MCMC adapts step size locally for better sampling efficiency.
problem Challenging step size selection for complex, multiscale targets.
method AutoStep MCMC uses a locally adaptive step size for involutive proposals.
result AutoStep MCMC is π-invariant, irreducible, and aperiodic.
This work studies scaling laws for low-precision training in high-dimensional linear regression.
problem Optimizing trade-off between model quality and training costs in high-dimensional linear regression.
method Theoretical study of scaling laws for low-precision training within a high-dimensional sketched linear regression framework, analyzing multiplicative and additive quantization.
result Multiplicative quantization maintains full-precision model size, while additive quantization reduces effective model size.
In this paper, we introduce a method for adapting the step-sizes of temporal difference (TD) learning. The performance of TD methods often depends on well chosen step-sizes, yet few algorithms have been developed for setting the step-size automatically for TD learning. An important limitation of current methods is that…
In this paper we address the question of the size distribution of firms. To this aim, we use the Bloomberg database comprising multinational firms within the years 1995-2003, and analyze the data of the sales and the total assets of the separate financial statement of the Japanese and the US companies, and make a compa…
This paper compares Transformers and RNNs in various tasks, showing size differences.
problem Comparing representational capabilities of Transformers and RNNs across tasks.
method Analysis of differences in tasks like index lookup, nearest neighbor, and string equality.
result Size differences in Transformers and RNNs for various tasks.
New analysis reveals batch size effects on stochastic conditional gradient methods.
problem Understanding the role of batch size in stochastic conditional gradient methods.
method Deriving a new analysis focusing on momentum-based stochastic conditional gradient algorithms (e.g., Scion).
result Increasing batch size initially improves optimization accuracy but can degrade performance beyond a critical threshold.
New scheme adapts batch size for faster variance-reduced algorithms.
problem Slowness of variance-reduced algorithms due to large batch size.
method Eliminates backtracking line search, adapts batch size via history stochastic gradients.
result Significantly reduces overall complexity for SVRG and SARAH/SPIDER.
Exchange improves liquidity by using different bid and ask tick sizes.
problem Improving liquidity quality in financial markets.
method Model with uncertainty zones to analyze bid and ask side-specific tick sizes.
result Suitable side-specific tick sizes enhance liquidity provision.
Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the …
New algorithm improves stability of optimization algorithms by adapting step-size.
problem Optimization algorithms' effectiveness is sensitive to step-size hyperparameters.
method Adapts NGN step-size method with momentum to enhance stability.
result Achieves convergence rate of O(1/√K) without restrictive assumptions.
Proposes a differentiable hypergeometric distribution for learning group importance.
problem Learning the sizes of subsets in applications like clustering and weakly-supervised learning.
method Introduces a reparameterizable hypergeometric distribution to model group sizes and learn their relative importance.
result Outperforms previous methods in weakly-supervised learning and clustering.
Adaptive step sizes improve optimization for convex and nonconvex problems.
problem Optimizing functions that are not strongly convex.
method Bridge nonconvex and strongly convex problems via regularization, then apply Barzilai-Borwein step sizes with SARAH.
result Regularized SARAH methods achieve better complexity in nonconvex problems.
Seesaw optimizes training by balancing learning rate and batch size, accelerating model pretraining.
problem Optimizing training efficiency for large language models with adaptive optimizers.
method Develops a principled framework for batch-size scheduling, introducing Seesaw which multiplies learning rate by 1/√2 and doubles batch size.
result Empirically, Seesaw reduces wall-clock time by approximately 36% compared to cosine decay, matching theoretical limits.
The paper analyzes the risk of bagging regularized M-estimators under proportional asymptotics.
problem Characterizing the risk of ensemble estimators trained with subsamples and regularizers.
method Developed a consistent estimator for the risk of ensemble estimators under proportional asymptotics.
result Optimal subsample size k⋆ tends to be in the overparameterized regime for the full-ensemble estimator.