Preprocessing data is an important step before any data analysis. In this paper, we focus on one particular aspect, namely scaling or normalization. We analyze various scaling methods in common use and study their effects on different statistical learning models. We will propose a new two-stage scaling method. First, w…
A new robust scaling approach improves downstream metabolomics analysis.
problem Challenges in choosing scaling techniques for metabolomics data.
method Introduces a weighted scaling approach robust to outliers.
result The proposed method outperforms traditional scaling techniques in both outlier-free and outlier-present datasets.
AutoScale improves LLM pre-training by adjusting data mixtures at different scales.
problem Data mixtures that work well at small scales may not perform as well at larger scales.
method AutoScale uses a two-stage approach: fitting a model to predict loss under different compositions and extrapolating optimal compositions to larger scales.
result AutoScale accelerates convergence and improves downstream performance.
This paper improves generative models by using data scaling and theoretical analysis.
problem Challenges in selecting noise distributions for stable learning in generative models.
method Introduces Scale-GAN, which uses data scaling and variance-based regularization.
result Data scaling controls the bias-variance trade-off and improves stability and accuracy.
Neural networks' performance scales with data size, explained by data manifold dimensionality.
problem Understanding the scaling of neural network performance with the number of parameters.
method Explained by the intrinsic dimension of the data manifold, confirmed through teacher/student framework and various datasets.
result The scaling exponent α is approximately 4 divided by the intrinsic dimension d of the data manifold.
This work explores variably scaled kernels to improve non-stationary Gaussian processes.
problem Limited ability of stationary kernels to represent heterogeneous correlation structures.
method Introduces variably scaled kernels to modify correlation structures explicitly.
result Improved reconstruction accuracy and better uncertainty estimates for non-stationary data.
Proposes a method to predict time series data using scale information.
problem Sequential data exhibits different behavior at different scales.
method Scale-based inference using k-nearest neighbors and sequential prediction. result Asymptotically, the algorithm produces the best prediction error.
New pruning method breaks power law scaling, potentially reducing error to exponential.
problem Improving neural network performance through scaling alone is costly.
method Developed a new data pruning metric to break power law scaling.
result Pruned datasets show better than power law scaling on various image datasets.
This paper improves conditional multidimensional scaling for incomplete data.
problem Handling missing data in known features for multidimensional scaling.
method Proposes a method to learn low-dimensional configurations with missing known feature values.
result Can learn low-dimensional configurations and impute missing values.
This paper reviews some of the phenomenological models which have been introduced to incorporate the scaling properties of financial data. It also illustrates a microscopic model, based on heterogeneous interacting agents, which provides a possible explanation for the complex dynamics of markets' returns. Scaling and m…
Analyzed US firm data 1970-2019, identifying scale effects and distributional forms.
problem Understanding differences between small and large firms over time.
method Examined all public US firms, used stylized facts and DLN distribution analysis.
result Small firms are systematically different from large firms, with scale-dependent heteroskedasticity.
Paper proposes data quality measures for large-scale high-dimensional data.
problem Lack of practical data quality measures for large-scale high-dimensional data.
method Proposes two data quality measures: class separability and in-class variability. Efficient algorithms based on random projections and bootstrapping are provided.
result Efficient algorithms for computing data quality measures on large-scale high-dimensional data.
The study explains transformer scaling laws using statistical and approximation theories.
problem Understanding why transformer scaling laws exist for large models trained on low-dimensional data.
method Established statistical estimation and mathematical approximation theories for transformers on low-dimensional manifolds.
result Predicted a power law between generalization error and model and data sizes, with power depending on intrinsic data dimension.
Improved loss scaling for stochastic momentum algorithms in high dimensions.
problem Improving loss scaling for stochastic momentum algorithms in high dimensions.
method Dimension-adapted Nesterov acceleration (DANA) scales momentum hyperparameters based on model size and data complexity.
result DANA improves loss scaling exponents across various data and target complexities.
Study how generalization scales with model size and data in quadratic neural networks.
problem Understanding how generalization scales with model size and data in quadratic neural networks.
method Analyzed ℓ2-regularized empirical test error minimization in a quadratic two-layer network with finite-sample setting and structured data. result Revealed a phase diagram with distinct scaling regimes as the number of parameters varies, showing data-dependent power laws controlled by spectral structure of the target.
Classical multidimensional scaling is an important dimension reduction technique. Yet few theoretical results characterizing its statistical performance exist. This paper provides a theoretical framework for analyzing the quality of embedded samples produced by classical multidimensional scaling. This lays the foundati…
Improved scaling laws in linear regression using data reuse.
problem Sustainability of neural scaling laws when running out of new data.
method Data reuse in multi-pass stochastic gradient descent (multi-pass SGD) for M-dimensional linear models trained on N data with sketched features. result Multi-pass SGD achieves a test error of Θ(M1−b+L(1−b)/a) with L>N, improving scaling laws in data-constrained regimes. CAST improves spectral clustering for multi-scale data by integrating reachability similarity.
problem Applying spectral clustering to multi-scale data where clusters vary in size and density.
method CAST integrates reachability similarity with distance-based similarity to derive a coefficient matrix, then applies trace Lasso regularization.
result CAST provides excellent performance and robustness across various multi-scale data test cases.
SIMPGEN improves SWOT SSH data interpretation by removing noise and preserving fine-scale features.
problem Noisy data and limited fine-scale observations in oceanic processes.
method Simulation-Informed Metric and Prior for Generative Ensemble Networks (SIMPGEN) combining real SWOT observations with simulated reference data.
result SIMPGEN effectively removes noise, preserving fine-scale features better than existing neural methods.
Paper presents an efficient algorithm for learning minimax risk classifiers with large-scale data.
problem Efficient learning of minimax risk classifiers for large-scale data with multiple classes.
method Combination of constraint and column generation for efficient learning.
result 10x speedup for general large-scale data and 100x speedup with many classes.
Unified scaling laws reveal how model size and training time impact neural network performance.
problem Understanding how much performance improvement can be expected from scaling model size or data volume.
method Established scale-time equivalence and combined it with a linear model analysis of double descent.
result Unified theoretical scaling laws explain previously unexplained phenomena and offer a more accessible path to training large models.
Theory predicts neural scaling exponents from language statistics.
problem No existing theory could quantitatively predict neural scaling exponents.
method Isolated two key statistical properties of language.
result Derives a simple formula predicting neural scaling exponents.
New phases identified in neural scaling laws with compute limits.
problem Understanding neural scaling laws under compute constraints.
method Solved neural scaling model with stochastic gradient descent, derived loss curves, analyzed model-parameter-count phases.
result Identified 4 phases (+3 subphases) in data-complexity/target-complexity phase-plane, derived exponents.
Study presents MMC model for better fitting multiple choice data.
problem Improving accuracy of latent trait estimates in IRT models.
method Fit autoencoders to MMC model, demonstrating better fit than nominal response model.
result MMC model outperforms traditional IRT models in fit.
Theory explains neural network scaling with dataset and model size.
problem Neural network scaling laws with dataset and model size.
method Identified variance-limited and resolution-limited scaling behaviors.
result Four scaling regimes explained: infinite data, infinite width, resolution-limited, and large width.
Spectral clustering algorithms typically require a priori selection of input parameters such as the number of clusters, a scaling parameter for the affinity measure, or ranges of these values for parameter tuning. Despite efforts for automating the process of spectral clustering, the task of grouping data in multi-scal…
We develop a scale-invariant truncated Lévy (STL) process to describe physical systems characterized by correlated stochastic variables. The STL process exhibits Lévy stability for the probability density, and hence shows scaling properties (as observed in empirical data); it has the advantage that all moments are fini…
We determine the critical batch size for large language models and find it scales with data size, not model size.
problem Determining the optimal batch size for large-scale model training.
method We propose a measure of critical batch size, pre-trained models, and systematic hyper-parameter sweeps.
result The critical batch size scales primarily with data size, not model size.
Efficiently trains deep Gaussian processes on large datasets.
problem Large-scale data and multi-scale features in function approximation.
method Combines variational learning with MCMC for efficient and accurate training.
result Highly efficient and accurate deep GP training on large-scale data.
Scaling laws in linear regression explain model performance improvements with size and data.
problem Disagreement between empirical neural scaling laws and conventional wisdom on variance error.
method Infinite dimensional linear regression setup, one-pass SGD, Gaussian prior, power-law spectrum.
result Variance error is dominated by other errors, disappearing from the bound due to SGD's implicit regularization.
Scaling laws govern predictive uncertainties in deep learning models.
problem Understanding predictive uncertainties in deep learning models.
method Empirical analysis and approximate Bayesian inference on vision and language tasks.
result Scaling laws exist for various measures of predictive uncertainty in deep learning models.
This work explains scaling laws as redundancy laws in deep learning.
problem The mathematical origins of scaling laws in deep learning models remain unclear.
method Kernel regression and analysis of data covariance spectra.
result Scaling laws can be explained as redundancy laws, revealing the learning curve's slope depends on data redundancy.
The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well…
New method makes quality metrics scale-invariant for high-dimensional data.
problem Scale sensitivity in quality metrics affects the accuracy of data projections.
method Analytical and empirical investigation of stress and KL divergence; introduction of a scale-invariant technique.
result The proposed technique accurately captures expected behavior and makes metrics scale-invariant.
This work analyzes neural scaling laws using power-law data spectra and derives analytical expressions for generalization error.
problem Understanding how neural network performance scales with key factors like data size and model complexity.
method Statistical mechanics techniques applied to one-pass stochastic gradient descent in a student-teacher framework.
result Derivation of analytical expressions for generalization error under power-law data spectra and identification of conditions for power-law scaling.
Study shows cliff-learning in transfer learning from foundation models.
problem Data-scaling of transfer learning from foundation models in low data regimes.
method Investigation of cliff-learning phenomenon through foundation-model analysis and toy models.
result Cliff-learning reflects compatibility between priors and tasks.
In this paper, we consider daily financial data of a collection of different stock market indices, exchange rates, and interest rates, and we analyze their multi-scaling properties by estimating a simple specification of the Markov-switching multifractal model (MSM). In order to see how well the estimated models captur…
We introduce online learning algorithms which are independent of feature scales, proving regret bounds dependent on the ratio of scales existent in the data rather than the absolute scale. This has several useful effects: there is no need to pre-normalize data, the test-time and test-space complexity are reduced, and t…
We introduce online learning algorithms which are independent of feature scales, proving regret bounds dependent on the ratio of scales existent in the data rather than the absolute scale. This has several useful effects: there is no need to pre-normalize data, the test-time and test-space complexity are reduced, and t…
This work introduces a protocol to automatically select the correct range of scales for meaningful Intrinsic Dimension estimation.
problem The Intrinsic Dimension (ID) varies with scale in real-world datasets, leading to erroneous results.
method The protocol selects the correct range of scales by ensuring constant density of data points.
result The method provides a robust and scale-adaptive approach to estimating meaningful Intrinsic Dimension.
Transformers can scale both context and task, but MLPs can only scale task.
problem Understanding and scaling In-Context Learning in transformers.
method Simplified transformer architecture, feature map, and MLP combination.
result Simplified transformer can perform ICL and context-scaling but not task-scaling.
Study uses CNNs to upscale wind speed data from 100 km to 3 km, improving subgrid-scale variability.
problem Recovering fine-scale wind speed information from coarse data.
method Convolutional neural networks (CNNs) with different input configurations (coarse wind speed, fine-scale topography, diurnal cycle) were tested.
result CNN models with coarse wind and fine topography inputs perform best in generalizing to unseen regions.
Early fault detection using instrumented sensor data is one of the promising application areas of machine learning in industrial facilities. However, it is difficult to improve the generalization performance of the trained fault-detection model because of the complex system configuration in the target diagnostic system…
Memory-efficient learning for large-scale imaging systems.
problem Memory limitations in GPUs for real-world large-scale inverse problems.
method Exploits reversibility of network layers to enable data-driven design.
result Demonstrated on small-scale and large-scale real-world systems.
A new, fast kernel test for large data.
problem Efficient kernel two-sample tests for high-dimensional, large-scale data.
method A new kernel-based test that is computationally efficient and robust to high dimensions.
result The new test performs well across various alternatives and dimensions.
CrossAD detects anomalies in time series data by considering cross-scale associations and cross-window modeling.
problem Anomaly detection in time series data is challenging due to varying patterns at different scales and fixed window sizes.
method CrossAD incorporates cross-scale reconstruction and a query library to capture dynamic cross-scale associations and comprehensive context.
result CrossAD achieves state-of-the-art performance in anomaly detection across multiple real-world datasets.
New kernel model scales to large datasets.
problem Challenges in scaling kernel machines to large datasets.
method Introduced EigenPro 3.0, an algorithm based on projected dual preconditioned SGD.
result Ability to scale model and data sizes independently.
Unified theory for neural scaling laws in hierarchically compositional data.
problem Understanding neural scaling laws in hierarchically compositional data.
method Probabilistic context-free grammars and power-law distributed production rules.
result Unified learning curve behavior for classification and next-token prediction tasks.