New method improves accuracy of compressed least-squares solutions.
problem Error and noise introduced by compression in least-squares problems.
method Proposes robust partially-compressed least-squares models and algorithms.
result Robust solutions more accurate than classical compressed variants.
A new framework for efficient large-scale learning using sketching of moments.
problem Efficiently learning from large datasets with limited computational resources.
method Compressing the training data into a low-dimensional sketch and solving a nonlinear least squares problem.
result Sufficient sketch sizes to control the generalization error of the procedure.
The paper proposes a least squares method for binary compressive sampling with low intrinsic dimension signals.
problem Recovering signals from binary measurements with noise and sign flips.
method Least squares decoder for signals with low generative intrinsic dimension.
result The least squares decoder achieves a sharp estimation error of O ( k log ( L n ) m ) O(\sqrt{\frac{k\log (Ln)}{m}}) O ( m k l o g ( L n ) ) under certain conditions. New techniques save bits in image compression with upsampling.
problem Lack of context dependence in current image compression methods with upsampling.
method Simple, inexpensive techniques exploiting context to predict Laplace distribution parameters.
result Average savings of 0.645 bits per difference, up to 1.489 bits.
Paper analyzes convergence rates of compressed LSR algorithms in federated learning.
problem Impact of compression on convergence rates in distributed learning.
method Analyzes a general stochastic approximation algorithm for LSR with weak assumptions.
result Convergence rates depend on the covariance of additive noise and compression strategy.
The paper proposes a control strategy for systems with sparse parameters using compressed sensing.
problem Control of linear systems with unknown sparse parameters under disturbances.
method Sparse estimation using Recursive Least Squares, improved with Basis Pursuit Denoising, and reformulated probabilistic constraints.
result The proposed algorithm outperforms existing methods in control design for systems with sparse impulse response parameters.
Recent research has studied the role of sparsity in high dimensional regression and signal reconstruction, establishing theoretical limits for recovering sparse models from sparse data. This line of work shows that ℓ 1 \ell_1 ℓ 1 -regularized least squares regression can accurately estimate a sparse linear model from n n n nois…
R2T hybrid model improves robust regression for asymmetric noise.
problem Least-squares regression fails with asymmetric structured noise.
method Transformer encoder, compression NN, fixed symbolic equation.
result Median regression MSE of 6e-6 to 3.5e-5 on synthetic data.
Efficiently price high-dimensional Bermudan options using tensor compression.
problem High-dimensional option pricing with computational complexity.
method Hierarchical tensor compression for Monte Carlo and dual martingale methods.
result Tensor compression alleviates the curse of dimensionality for Bermudan option pricing.
In this paper, we study a fast approximation method for {\it large-scale high-dimensional} sparse least-squares regression problem by exploiting the Johnson-Lindenstrauss (JL) transforms, which embed a set of high-dimensional vectors into a low-dimensional space. In particular, we propose to apply the JL transforms to …
In this paper we study the performance of the Projected Gradient Descent(PGD) algorithm for ℓ p \ell_{p} ℓ p -constrained least squares problems that arise in the framework of Compressed Sensing. Relying on the Restricted Isometry Property, we provide convergence guarantees for this algorithm for the entire range of $0\leq p\…
New compression methods outperform FEFA in ASV tasks with similar accuracy but significantly faster.
problem Improving ASV system development speed while maintaining accuracy.
method Compared and evaluated several supervector compression methods including PPCA, FA, SPPCA, and PPLS.
result Supervector compression approaches are as effective as FEFA in ASV tasks but offer significant speed improvements.
A new method extends ridge regression with stretchable parameters for high-dimensional data.
problem Handling high-dimensional data for effective learning.
method Introduces ridge regression with stretchable parameters and provides closed-form solutions.
result Empirical evaluations show the method is effective for compressive learning with high-dimensional data.
Over the past few years, trace regression models have received considerable attention in the context of matrix completion, quantum state tomography, and compressed sensing. Estimation of the underlying matrix from regularization-based approaches promoting low-rankedness, notably nuclear norm regularization, have enjoye…
We develop a primal dual active set with continuation algorithm for solving the \ell^0-regularized least-squares problem that frequently arises in compressed sensing. The algorithm couples the the primal dual active set method with a continuation strategy on the regularization parameter. At each inner iteration, it fir…
New faster, space-saving methods for subspace embeddings in tensors.
problem Efficiently embedding large tensors with fewer random bits.
method Modewise Johnson-Lindenstrauss embeddings for rank- r r r tensors. result Improved space complexity for tensor subspaces with fewer random bits.
Generalization in nonlinear least squares can be studied via algorithmic stability and effective dimension.
problem Generalization in nonlinear least squares models
method Deriving error bounds for local minimizers using algorithmic stability and effective dimension
result Bounds depend on learned geometry rather than parameter count
Efficient distributed learning with Byzantine-resilient thresholding and error feedback.
problem Byzantine-resilient distributed learning with communication efficiency.
method Simple thresholding for Byzantine mitigation, compressed gradients and norms for aggregation, error feedback.
result Statistical error rate matches Yin et al.~\cite{dong} but with simpler schemes, and improved convergence with error feedback.
Article presents QR and LQ decomposition algorithms for various matrix sizes and ranks.
problem Solving least squares problems in machine learning and computer vision.
method Developed novel matrix backpropagation algorithms for QR and LQ decompositions of different matrix sizes and ranks.
result Numerical stability and computational efficiency of the proposed methods.
The paper analyzes AR model estimation with limited data, showing improved sampling requirements.
problem Estimating AR model parameters from a small number of samples.
method Analyzes ℓ 1 \ell_1 ℓ 1 -regularized least squares and greedy estimators, characterizing sampling trade-offs. result Stable AR parameter recovery is possible with sub-linearly scaled samples, improving over existing methods.
The ℓ \ell ℓ -1 norm based optimization is widely used in signal processing, especially in recent compressed sensing theory. This paper studies the solution path of the ℓ \ell ℓ -1 norm penalized least-square problem, whose constrained form is known as Least Absolute Shrinkage and Selection Operator (LASSO). A solution path …
Algorithm estimates sparse signals from linear measurements, improving recovery guarantees.
problem Estimating gradient-sparse signals from noisy linear measurements.
method Iterative alpha expansion with proximal descent and geometric penalty decay.
result Global recovery guarantees under cut-restricted isometry property for Gaussian designs.
GANCS uses GANs to speed up MRI reconstruction while maintaining diagnostic quality.
problem Time and resource intensive MRI reconstruction and loss of diagnostic quality in compressed sensing.
method Generative adversarial networks (GAN) trained on historical patient data to learn diagnostic-quality MR images.
result GANCS reconstructs MRI images in a few milliseconds with high contrast and texture details.
We consider solving the ℓ 1 \ell_1 ℓ 1 -regularized least-squares ( ℓ 1 \ell_1 ℓ 1 -LS) problem in the context of sparse recovery, for applications such as compressed sensing. The standard proximal gradient method, also known as iterative soft-thresholding when applied to this problem, has low computational cost per iteration but a r…
New method uses deep learning to solve linear inverse problems.
problem Solving linear inverse problems with high-dimensional signals.
method Stochastic coarse-to-fine gradient ascent procedure using implicit prior from denoising CNN.
result General algorithm for solving linear inverse problems without additional training.
CD converges linearly for MCP/SCAD penalized least squares.
problem Recovering sparse signals from data.
method Coordinate descent for MCP/SCAD penalized least squares.
result CD converges linearly to solutions of MCP/SCAD penalized least squares.
Connection found between signal processing and slime mold dynamics.
problem Convergence of IRLS algorithm remains an open problem.
method Connection between IRLS and Physarum dynamics, proving convergence of a damped version.
result Convergence and complexity bounds for a damped version of the IRLS algorithm.
New method corrects least-squares temporal difference for better lambda-return estimation.
problem Improving lambda-return estimation in reinforcement learning.
method Uncorrected least-squares temporal difference with a correction method.
result Enhanced accuracy in temporal difference learning.
Overparameterization aids in model pruning, leading to improved test accuracy.
problem Improving lightweight model performance through pruning.
method Theoretical analysis and high-dimensional asymptotics of model pruning in overparameterized neural networks.
result Even with known informative features, training a large model and then pruning leads to better test accuracy.
Illustrates interleaved learning with Kalman Filter for linear least squares.
problem Improving machine learning algorithms through interleaved learning.
method Simple statistical and optimization framework based on Kalman Filter.
result Demonstrates the effectiveness of interleaved learning.
We study randomized sketching methods for approximately solving least-squares problem with a general convex constraint. The quality of a least-squares approximation can be assessed in different ways: either in terms of the value of the quadratic objective function (cost approximation), or in terms of some distance meas…
Cross validation residuals extended to GLS models.
problem Validating models with correlated data.
method Leave-M-out cross validation for GLS models, demonstrating relationship with Cook's distance.
result No need to refit model for reduced datasets.
We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that …
Flexible framework compresses models using LC algorithm.
problem Efficiently compressing neural networks for resource constraints.
method Decouples learning and compression steps with alternating L and C phases.
result Compressed models maintain performance and accuracy.
Reduces multiclass and regression compression schemes to binary ones.
problem Developing efficient learning algorithms for multiclass and regression problems.
method Reduces sample compression schemes for binary classes to multiclass and regression settings.
result Establishes new compression schemes for multiclass and regression problems.
Proposes a link between randomness and compression in deep learning.
problem Improving efficiency in deep learning training.
method Introduces a novel tomographic compression framework called Dual Tomographic Compression (DTC).
result Demonstrates high correlation between learning performance and Gibbs entropy over compression ratios.
A new method classifies images using a compressed summary.
problem Classifying images efficiently.
method Compressive learning with a novel sketch function.
result Improved image classification with less data.
Unified framework for non-compressed networks from compression bounds.
problem Generalization of large deep neural networks with huge model size.
method Unified framework converting compression based bounds to non-compressed networks.
result Unified data-dependent generalization error bound with tighter evaluation.
PARSEC compresses text for sentiment analysis with minimal loss in accuracy.
problem Compressing text data for sentiment analysis without losing accuracy.
method Uses Parts-of-Speech tags to compress text intelligently.
result Accurate compression is possible with minimal loss in sentiment classification accuracy.
Paper introduces a new adaptive gradient method with gradient compression for distributed training.
problem Communication overhead in distributed machine learning systems.
method Adaptive gradient method with gradient compression, scalable system BytePS-Compress.
result Convergence rate of O ( 1 / T ) \mathcal{O}(1/\sqrt{T}) O ( 1/ T ) for non-convex problems. Generalizes model compression as constrained optimization, including various techniques.
problem Large neural nets for tasks like object recognition are too big for mobile devices.
method Formulated as constrained optimization, developed a general algorithm using augmented Lagrangian and alternating optimization.
result Guaranteed to find the best compressed model for the task locally.
Galen algorithm compresses neural networks for specific hardware with reduced latency.
problem Finding optimal compression policies for neural networks on specific hardware.
method Reinforcement learning using pruning and quantization to optimize inference latency.
result Compressed ResNet18 for ARM processor reduced inference latency by 80%.
The (isothermic) compressibility of lattice knots can be examined as a model of the effects of topology and geometry on the compressibility of ring polymers. In this paper, the compressibility of minimal length lattice knots in the simple cubic, face centered cubic and body centered cubic lattices are determined. Our r…
Simplified proof shows SGD optimality for least squares.
problem Optimizing SGD for least squares efficiency.
method Analyzing SGD as a stochastic process, characterizing stationary covariance matrix.
result Statistical minimax optimality of SGD for least squares.
The paper improves Kaczmarz algorithm with momentum for linear least squares.
problem Improving convergence of the Kaczmarz algorithm for linear least squares.
method Integrates geometrically smoothed momentum into the randomized Kaczmarz algorithm.
result Proves expected error reduction in singular vector directions.
New algorithm improves online binary classification with constant time complexity.
problem Online binary classification with rebalancing.
method Non-iteratively reweighted recursive least-squares.
result Exacts converges to batch formulation and outperforms existing algorithms.
Paper compresses neural network weight-updates for image artifacts removal.
problem Efficiently compressing neural network weight-updates for image artifacts removal.
method Fine-tuning a pre-trained artifact removal network on target data with a compression objective that encourages sparse and quantized weight-updates.
result Achieves reconstruction quality comparable to traditional codecs at comparable bitrates.
HiLLoC compresses large images losslessly using VAEs.
problem Lossless compression of large color photographs.
method Fully convolutional VAE models trained on ImageNet are applied to lossless compression.
result Achieves state-of-the-art compression for full-size ImageNet images.