Adapts DR objectives for both sample and feature size reduction.
problem Simultaneously reduce sample and feature sizes.
method Semi-relaxed Gromov-Wasserstein optimal transport.
result OT plan delivers competitive hard clustering.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Model size reduction”
Paper reduces recommender system model size by 90%.
problem Large model size in recommender systems.
method Hybrid frequency and double hashing for model size reduction.
result 90% reduction in model size with no performance loss.
In this paper we propose the macroblock scaling (MBS) algorithm, which can be applied to various CNN architectures to reduce their model size. MBS adaptively reduces each CNN macroblock depending on its information redundancy measured by our proposed effective flops. Empirical studies conducted with ImageNet and CIFAR-…
Improved variance reduction for Riemannian non-convex optimization with adaptive batch size.
problem Optimizing non-convex functions on Riemannian manifolds.
method Batch size adaptation in R-SVRG, R-SRG, and R-SPIDER.
result Achieves lower total complexities for various non-convex functions.
Value selection reduces model size while maintaining accuracy.
problem Space efficiency in model size reduction.
method Two probabilistic methods based on information theory's metric: PVS and P + VS.
result Value selection achieves balance between accuracy and model size reduction.
Study finds that only a fraction of data is needed for accurate patient-level prediction models.
problem Developing predictive models for patient-level outcomes using large observational data.
method Empirical assessment of sample size effects on model performance and complexity using learning curves.
result A median reduction of 9.5% to 78.5% in the number of observations and 8.6% to 68.3% in the number of predictors can be achieved with adequate sample size.
Similar to convolution neural networks, recurrent neural networks (RNNs) typically suffer from over-parameterization. Quantizing bit-widths of weights and activations results in runtime efficiency on hardware, yet it often comes at the cost of reduced accuracy. This paper proposes a quantization approach that increases…
Pruning improves model generalization in over-parameterized models, contradicting traditional theories.
problem Pruning's effect on generalization in over-parameterized models.
method Empirical study on standard pruning algorithms and additional regularization effects.
result Pruning leads to better training and regularization, improving generalization.
Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatchcs.LG
New method ensures consistent inference across different tensor parallel sizes for large language models.
problem Non-deterministic inference in large language models due to inconsistent reduction orders across GPUs.
method Tree-Based Invariant Kernels (TBIK) that align intra- and inter-GPU reduction orders through a unified hierarchical binary tree structure.
result Bit-wise identical results across different tensor parallel sizes for RL training.
Paper proposes a method to estimate variance reduction in DNN training using importance sampling.
problem Challenges in assessing variance reduction during DNN training using importance sampling.
method Proposes a method for estimating variance reduction using minibatches sampled under importance sampling.
result Demonstrates consistent reduction in variance, improved training efficiency, and enhanced model accuracy.
SPreVcs.GR
SPREV simplifies visualization of complex labeled datasets.
problem Challenges of reducing dimensions and visualizing labeled datasets with small class size, high dimensionality, and low sample size.
method SPREV uses a novel dimensionality reduction technique integrating geometric principles.
result SPREV effectively visualizes hidden patterns in complex labeled datasets.
In this paper we study a family of variance reduction methods with randomized batch size---at each step, the algorithm first randomly chooses the batch size and then selects a batch of samples to conduct a variance-reduced stochastic update. We give the linear convergence rate for this framework for composite functions…
This paper explores how effective sample size, dimensionality, and model performance are related in covariate shift adaptation.
problem Understanding the relationship between effective sample size, dimensionality, and generalization in covariate shift adaptation.
method Building a unified theory connecting effective sample size, data dimensionality, and generalization in the context of covariate shift adaptation.
result Dimensionality reduction or feature selection can increase effective sample size, supporting the practice of reducing dimensionality before covariate shift adaptation.
Develops a new model to predict training dynamics of large language models.
problem Lack of mechanistic understanding of training dynamics in large language models.
method A first-principles reduced-order model of training dynamics, predicting group-size invariance and stability thresholds.
result Closed-form model predicts training dynamics with high accuracy and provides new diagnostics.
Approximates large Random Forest models to save space.
problem Large model size limits deployment on small devices.
method Approximate each tree using multinomial logistic regression.
result Reduced model size without significant loss of accuracy.
IMPACT optimizes LLM compression by focusing on activation importance, reducing model size up to 55.4%.
problem Resource constraints in deploying large language models (LLMs).
method IMPACT integrates activation importance into low-rank compression, optimizing for both size and accuracy.
result IMPACT achieves up to 55.4% greater model size reduction while maintaining comparable or better accuracy.
The variance reduction class of algorithms including the representative ones, SVRG and SARAH, have well documented merits for empirical risk minimization problems. However, they require grid search to tune parameters (step size and the number of iterations per inner loop) for optimal performance. This work introduces `…
The main goal of this work is equipping convex and nonconvex problems with Barzilai-Borwein (BB) step size. With the adaptivity of BB step sizes granted, they can fail when the objective function is not strongly convex. To overcome this challenge, the key idea here is to bridge (non)convex problems and strongly convex …
New RL algorithm reduces sample complexity for optimal learning.
problem Achieving optimal learning with minimal samples in RL.
method Early-settled variance reduction method with Q-learning sequences.
result Near-optimal regret achieved with sample size .
Reduced reservoir size for faster edge computing.
problem Efficiently reducing computational resources for reservoir computing.
method Concatenating past or drifting states of the reservoir to the output layer.
result Reduced reservoir size up to one tenth without significant error increase.
Paper proposes a method to use in silico experiments with foundation models to reduce sample size.
problem Costly and uncertain randomized experiments.
method Integrates predictions from multiple foundation models with experimental data.
result Estimator offers substantial precision gains, equivalent to a 20% reduction in sample size.
Knowledge distillation (KD) is a very popular method for model size reduction. Recently, the technique is exploited for quantized deep neural networks (QDNNs) training as a way to restore the performance sacrificed by word-length reduction. KD, however, employs additional hyper-parameters, such as temperature, coeffici…
In most OTC markets, a small number of market makers provide liquidity to other market participants. More precisely, for a list of assets, they set prices at which they agree to buy and sell. Market makers face therefore an interesting optimization problem: they need to choose bid and ask prices for making money while …
Stochastic gradient Markov Chain Monte Carlo (SG-MCMC) has been developed as a flexible family of scalable Bayesian sampling algorithms. However, there has been little theoretical analysis of the impact of minibatch size to the algorithm's convergence rate. In this paper, we prove that under a limited computational bud…
A new dimension reduction method based on Gaussian finite mixtures is proposed as an extension to sliced inverse regression (SIR). The model-based SIR (MSIR) approach allows the main limitation of SIR to be overcome, i.e., failure in the presence of regression symmetric relationships, without the need to impose further…
Algorithme EM régulariséstat.ML
Regularized EM algorithm improves clustering performance with small sample sizes.
problem Performance reduction in EM algorithm due to small sample size and poorly conditioned covariance matrices.
method Regularized EM algorithm that uses prior knowledge to ensure positive definiteness of covariance matrices.
result The regularized EM algorithm outperforms standard EM in clustering tasks with small sample sizes.
A new layer, funnel, reduces dimensionality in flows for better performance.
problem Training high-dimensional models efficiently and accurately.
method Constructing dimension-reducing surjective flows using the funnel layer.
result The funnel layer improves model performance with a smaller latent space.
Machine learning-based applications are increasingly prevalent in IoT devices. The power and storage constraints of these devices make it particularly challenging to run modern neural networks, limiting the number of new applications that can be deployed on an IoT system. A number of compression techniques have been pr…
Improved recommendation systems using multi-layer embeddings reduce model size while maintaining accuracy.
problem Improving model accuracy in recommendation systems while minimizing model size.
method Introducing a multi-layer embedding training (MLET) architecture that trains embeddings via a sequence of linear layers.
result Substantial advantages in model accuracy and memory footprint are achieved with reduced embedding dimensions.
How can we train a statistical mixture model on a massive data set? In this work we show how to construct coresets for mixtures of Gaussians. A coreset is a weighted subset of the data, which guarantees that models fitting the coreset also provide a good fit for the original data set. We show that, perhaps surprisingly…
Convolution Neural Networks (CNN) have performed well in many applications such as object detection, pattern recognition, video surveillance and so on. CNN carryout feature extraction on labelled data to perform classification. Multi-label classification assigns more than one label to a particular data sample in a data…
Revisits granular models explaining firm growth rates and sizes.
problem Understanding the relationship between firm size and growth rate statistics.
method Developed new theoretical insights linking firm size and growth rate statistics within granular models.
result Growth volatility distribution is size-independent but fat-tailed, challenging granular models.
In the past decade, sparse principal component analysis has emerged as an archetypal problem for illustrating statistical-computational tradeoffs. This trend has largely been driven by a line of research aiming to characterize the average-case complexity of sparse PCA through reductions from the planted clique (PC) con…
We propose a new yet natural algorithm for learning the graph structure of general discrete graphical models (a.k.a. Markov random fields) from samples. Our algorithm finds the neighborhood of a node by sequentially adding nodes that produce the largest reduction in empirical conditional entropy; it is greedy in the se…
Machine learning reduces combinatorial optimization problem dimensions.
problem Reducing the complexity of large combinatorial optimization problems.
method Generalization of a machine learning model for problem reduction on TSP.
result Machine learning can predict which variables are not part of an optimal solution.
U-statistics improve gradient estimation in importance-weighted variational inference.
problem High variance in gradient estimation for importance-weighted variational inference.
method Use U-statistics to average base gradient estimators on overlapping batches of size m, achieving lower variance.
result U-statistic variance reduction leads to modest to significant improvements in inference performance.
A new RNN architecture reduces model size and improves performance.
problem Overparameterization and resource limitations in RNNs.
method Jointly encodes weight matrices using tensor-train factorization.
result Reduces model size by several orders of magnitude without sacrificing performance.
Develops a new SPP algorithm with variance reduction for weakly convex optimization.
problem Weakly convex, composite optimization problems.
method Inexact semismooth Newton framework with variance reduction for stochastic proximal point updates.
result Establishes convergence results for the proposed algorithm.
This paper compares Transformers and RNNs in various tasks, showing size differences.
problem Comparing representational capabilities of Transformers and RNNs across tasks.
method Analysis of differences in tasks like index lookup, nearest neighbor, and string equality.
result Size differences in Transformers and RNNs for various tasks.
Study compares under-bagging with other methods for imbalanced data.
problem Comparing under-bagging with other methods for imbalanced data.
method Replica analysis of under-bagging, comparing with under-sampling and simple weighting.
result Under-bagging improves performance by increasing the majority class size.
A new algorithm reduces the size of datasets for TDA.
problem Processing large datasets with high dimensions in TDA is computationally infeasible.
method Introduced Characteristic Lattice Algorithm (CLA) for data reduction.
result CLA reduces dataset size while preserving geometric and topological features.
New method improves convergence and reduces variance in noisy optimization problems.
problem Computing exact minimizers with noisy gradient information.
method Stochastic mirror descent with interacting particles.
result Interaction helps improve convergence and reduce variance.
We investigate the statistical properties of the EBS order book for the EUR/USD and USD/JPY currency pairs and the impact of a ten-fold tick size reduction on its dynamics. A large fraction of limit orders are still placed right at or halfway between the old allowed prices. This generates price barriers where the best …
Can one reduce the size of a graph without significantly altering its basic properties? The graph reduction problem is hereby approached from the perspective of restricted spectral approximation, a modification of the spectral similarity measure used for graph sparsification. This choice is motivated by the observation…
We propose a novel method to train deep convolutional neural networks which learn from multiple data sets of varying input sizes through weight sharing. This is an advantage in chemometrics where individual measurements represent exact chemical compounds and thus signals cannot be translated or resized without disturbi…
This work studies the implicit bias of mini-batch SGD in classification.
problem Understanding the implicit bias of mini-batch SGD in multi-class classification.
method Characterizes how batch size, momentum, and variance reduction affect convergence and max-margin behavior under different norms.
result Momentum enables small-batch convergence to an approximate max-margin solution, while variance reduction recovers the exact full-batch bias.
New algorithm reduces optimization complexity in adaptive mirror descent.
problem Optimizing complex, non-smooth, non-convex functions efficiently.
method SVRAMD: Variance Reduced Adaptive Mirror Descent.
result Variance reduction accelerates convergence in adaptive mirror descent.
How to Scale Your EMAstat.ML
This work provides a scaling rule for model EMA optimization across batch sizes.
problem Training dynamics and performance differences across batch sizes when using model EMA.
method Developed a scaling rule for model EMA optimization, demonstrating its validity across various architectures and data modalities.
result Enabled SSL methods like BYOL to train at larger batch sizes without performance degradation.