First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity. Second-order methods, while able to provide faster convergence, have been much less explored due to the high cost of computing the second-order information. In this paper we …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Second order sparsity”
New theory allows ICA without assuming non-Gaussian sources.
problem Traditional ICA struggles with Gaussian sources.
method Developed identifiability theory based on second-order statistics and sparsity.
result Identifiability theory and estimation methods validated experimentally.
Paper introduces structured sparsity estimators for Generalized Linear Models.
problem Estimating structured sparsity in GLMs with debiased estimators.
method Extends Stucky and van de Geer's results to GLMs with structured sparsity.
result Proves oracle inequalities for structured sparsity estimators in GLMs.
A new method solves large-scale sparse group square-root Lasso problems efficiently.
problem Large-scale linearly constrained sparse group square-root Lasso problems.
method Dual semismooth Newton based augmented Lagrangian method (ALM).
result The proposed method efficiently solves the problem with numerical experiments demonstrating its effectiveness.
In this paper, we propose -norm regularized models to seek near-optimal sparse portfolios. These sparse solutions reduce the complexity of portfolio implementation and management. Theoretical results are established to guarantee the sparsity of the second-order KKT points of the -norm regularized models…
Paper proposes a new sparsity scheme for high-dimensional VAR models.
problem Estimation of high-dimensional VAR models with sparsity assumptions.
method Regularized estimation procedures for sparse VAR models.
result Threholding extends consistency properties of regularized estimators.
New regularization scheme for FMs improves feature interaction selection.
problem Feature selection in FMs leads to loss of feature interactions.
method Proposes a new regularization scheme for FMs with upper bound of regularizer.
result Improves feature interaction selection without restricting sparsity patterns.
The paper deals with the problem of finding sparse solutions to systems of polynomial equations possibly perturbed by noise. In particular, we show how these solutions can be recovered from group-sparse solutions of a derived system of linear equations. Then, two approaches are considered to find these group-sparse sol…
New methods show sparse portfolios offer no advantage over mean-variance in diversification.
problem Investment diversification and risk management with sparse portfolios.
method Developed and implemented a new estimation procedure for sparse second-order stochastic spanning using a greedy algorithm and Linear Programming.
result No benefit from expanding a sparse opportunity set beyond 45 assets; optimal sparse portfolio reduces tail risk.
In this paper, we propose and analyze zeroth-order stochastic approximation algorithms for nonconvex and convex optimization, with a focus on addressing constrained optimization, high-dimensional setting and saddle-point avoiding. To handle constrained optimization, we first propose generalizations of the conditional g…
New method identifies causal order without sparsity assumptions.
problem Causal order discovery in observational data.
method Sequential procedure to directly identify causal order.
result Direct identification of causal order without sparsity assumptions.
Paper introduces a novel matrix-wise sparse MNNLS formulation and algorithm.
problem Sparse nonnegative least squares with multiple right-hand sides.
method Matrix-wise sparsity constraint, two-step algorithm.
result More accurate results compared to state-of-the-art methods.
Efficiently solves Elastic Net in high dimensions with Newton method.
problem Feature selection in high-dimensional data with non-negligible collinearity.
method Semi-smooth Newton Augmented Lagrangian Method.
result Significantly reduces computational cost compared to competitors.
New method computes affine normal directions efficiently for sparse polynomials.
problem Computing affine normal directions is computationally expensive in high dimensions.
method Reduces third-order tensor contraction to matrix-free formulation using log-determinant gradient.
result Scalable implementations with near-linear scaling in dimension and sparsity.
Variable selection is one of the most important tasks in statistics and machine learning. To incorporate more prior information about the regression coefficients, the constrained Lasso model has been proposed in the literature. In this paper, we present an inexact augmented Lagrangian method to solve the Lasso problem …
SpeqNets improve graph neural networks by scaling and adapting to graph sparsity.
problem Graph neural networks struggle with permutation-equivariant functions and scalability to large graphs.
method Introducing sparsity-aware, permutation-equivariant graph networks with heuristics for graph isomorphism.
result Significantly improved predictive performance and reduced computation times compared to existing methods.
Robust STAP with coprime arrays reduces clutter using sparse modeling.
problem Limited performance due to training samples support in practical applications.
method Two-stage approach: 1) RD virtual snapshot, 2) RD sparse measurement modeling with OMP-like recovery.
result Robust to prior knowledge errors, good clutter suppression performance.
We propose to optimize the activation functions of a deep neural network by adding a corresponding functional regularization to the cost function. We justify the use of a second-order total-variation criterion. This allows us to derive a general representer theorem for deep neural networks that makes a direct connectio…
Improved Compressed Sensing by optimizing sparse solutions with mixed integer programming.
problem Finding sparse solutions to linear measurements with numerical tolerance.
method Introducing an regularized formulation, reformulating as a mixed integer second order cone program, deriving a second order cone relaxation, and developing a custom branch-and-bound algorithm.
result Our approach produces solutions that are on average 6.22% more sparse compared to state-of-the-art methods.
Randomized feature models learn interaction kernels from agent paths.
problem Learning interaction kernels from noisy agent paths.
method Randomized feature algorithm and sparse regression.
result Pruned features reduce overfitting and lower simulation cost.
MOMENT selects and estimates mixed-effects models using moment identities.
problem Selecting and estimating random-effects covariance matrix and fixed-effects coefficients in multiresponse linear mixed-effects models.
method MOMENT is a stage-wise moment-based framework that reduces the random-effects selection problem to a smooth constrained convex optimization problem.
result MOMENT performs competitively and can outperform separate univariate analyses for correlated responses.
Study sparsity benefits in infinite feature contextual bandits.
problem Minimizing regret in infinite feature contextual bandits.
method Novel reduction to multi-armed bandits, Feel-Good Thompson Sampling algorithm.
result Regret bounds match lower bounds up to logarithmic factors, logarithmic dependence on effective features.
Popular sparse estimation methods based on -relaxation, such as the Lasso and the Dantzig selector, require the knowledge of the variance of the noise in order to properly tune the regularization parameter. This constitutes a major obstacle in applying these methods in several frameworks---such as time series, …
Paper proves higher-order flow matching preserves optimality in generative modeling.
problem Theoretical guarantees for higher-order flow matching in generative modeling.
method Neural network approximations with controlled depth, width, and sparsity.
result Proves worst case optimality for second-order flow matching.
Generalized notions of sparsity and restricted isometry property. Part I: A unified frameworkstat.ML
The restricted isometry property (RIP) is an integral tool in the analysis of various inverse problems with sparsity models. Motivated by the applications of compressed sensing and dimensionality reduction of low-rank tensors, we propose generalized notions of sparsity and provide a unified framework for the correspond…
A new method for embedding sparse high-order interactions.
problem Learning embeddings from sparse high-order interaction events.
method Hybridizing sparse hypergraph and matrix Gaussian processes.
result Strong asymptotic bounds on sparsity ratio.
New method solves sparse PCA for multiple components efficiently.
problem Sparse PCA for multiple orthogonal components.
method Reformulates orthogonality as rank constraints, uses semidefinite relaxations and bounds.
result Exact solutions with near-optimal variance explained and orthogonality.
Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as w…
FlipOut prunes neural networks by flipping weights' signs, achieving high sparsity.
problem Redundant weights in neural networks increase training time and resource usage.
method Uses sign flips during training to determine weight saliency for pruning.
result Competitive with existing methods, achieving state-of-the-art performance for high sparsity.
Principal components analysis (PCA) is the optimal linear auto-encoder of data, and it is often used to construct features. Enforcing sparsity on the principal components can promote better generalization, while improving the interpretability of the features. We study the problem of constructing optimal sparse linear a…
Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretationcs.LG
Current methods to interpret deep learning models by generating saliency maps generally rely on two key assumptions. First, they use first-order approximations of the loss function neglecting higher-order terms such as the loss curvatures. Second, they evaluate each feature's importance in isolation, ignoring their int…
The paper explores how multiway data from PDEs can be accurately tracked using EnKF with specific covariance and precision estimators.
problem Tracking sparse and multiway structures in dynamical processes governed by PDEs.
method Examined several multiway covariance and precision matrix estimators in the context of physics-driven forecasting and EnKF.
result Multiway data from Poisson and convection-diffusion PDEs can be accurately tracked using EnKF with appropriate estimators.
Several Approximation Algorithms for Sparse Best Rank-1 Approximation to Higher-Order Tensorsmath.NA
Four algorithms improve sparse tensor BR1Approx with theoretical guarantees.
problem Sparse tensor best rank-1 approximation.
method Four approximation algorithms exploiting multilinearity and sparsity.
result Theoretical worst-case approximation lower bounds for all algorithms.
E-Commerce (E-Com) search is an emerging important new application of information retrieval. Learning to Rank (LETOR) is a general effective strategy for optimizing search engines, and is thus also a key technology for E-Com search. While the use of LETOR for web search has been well studied, its use for E-Com search h…
The restricted isometry property (RIP) is a universal tool for data recovery. We explore the implication of the RIP in the framework of generalized sparsity and group measurements introduced in the Part I paper. It turns out that for a given measurement instrument the number of measurements for RIP can be improved by o…
Compressive sensing (CS) exploits sparsity to recover sparse or compressible signals from dimensionality reducing, non-adaptive sensing mechanisms. Sparsity is also used to enhance interpretability in machine learning and statistics applications: While the ambient dimension is vast in modern data analysis problems, the…
New algorithm reduces dimensionality in stochastic optimization.
problem Stochastic optimization in high-dimensional problems.
method Proposes a sparsity-inducing stochastic gradient-free (SI-SGF) algorithm.
result Proves dimension-free query complexity in convex and strongly convex cases.
The paper proposes a control strategy for systems with sparse parameters using compressed sensing.
problem Control of linear systems with unknown sparse parameters under disturbances.
method Sparse estimation using Recursive Least Squares, improved with Basis Pursuit Denoising, and reformulated probabilistic constraints.
result The proposed algorithm outperforms existing methods in control design for systems with sparse impulse response parameters.
We study a generalized framework for structured sparsity. It extends the well-known methods of Lasso and Group Lasso by incorporating additional constraints on the variables as part of a convex optimization problem. This framework provides a straightforward way of favouring prescribed sparsity patterns, such as orderin…
We demonstrate that, in the classical non-stochastic regret minimization problem with decisions, gains and losses to be respectively maximized or minimized are fundamentally different. Indeed, by considering the additional sparsity assumption (at each stage, at most decisions incur a nonzero outcome), we derive…
Double Machine Learning estimators are asymptotically inadmissible under structure-agnostic models.
problem Minimax estimators may be inadmissible under structure-agnostic models.
method Exhibit second-order (U-statistic) estimators that asymptotically dominate DML estimators.
result Double Machine Learning estimators are asymptotically inadmissible under structure-agnostic models.
Proposes a sparsity algorithm to improve corporate credit ratings.
problem Improving credit ratings of publicly traded companies.
method Formulates counterfactual explanation as an optimization problem and proposes a sparsity algorithm to maximize sparsity.
result The sparsity algorithm can capture features that improve credit ratings.
What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorystat.ML
New framework explains deep neural networks using variational spline theory.
problem Understanding functions learned by deep neural networks.
method Developed a variational framework and function space.
result Deep ReLU networks are solutions to regularized data fitting problems over the proposed function space.
New control theory shows neural networks can be sparsely active over time.
problem Optimizing neural networks for long-time control with sparsity constraints.
method Proving optimal controls vanish after a positive time and providing a stability estimate.
result Optimal controls for -penalized neural ODEs are sparsely active over time.
CoDeQ simplifies joint model compression by integrating pruning and quantization.
problem Joint pruning and quantization methods are complex and require additional procedures.
method CoDeQ uses a dead-zone quantizer to directly induce sparsity and learn quantization parameters.
result CoDeQ achieves high sparsity and low-precision accuracy with minimal bit operations.
Paper tackles DP-SCO with heavy-tailed data in high dimensions, improving error bounds.
problem Differentially private stochastic optimization with heavy-tailed data in high-dimensional spaces.
method Proposes methods for DP-SCO with polytope constraints and LASSO, analyzing sparsity constraints.
result Achieved near optimal error bounds for DP-SCO with heavy-tailed data.
New method reduces PDE model parameters by 30% with sparsity.
problem Redundant parameters in neural network projections.
method Bregman iterations for sparsity, POD compression, bias propagation.
result 30% fewer parameters with similar accuracy.
Second-order economic theory considers new variables to improve price volatility predictions.
problem Current economic models focus on first-order variables, missing second-order variables that affect price volatility.
method Introduces second-order economic theory with new variables composed of sums of squares of agents' transactions.
result Second-order economic theory complements first-order variables and introduces new macroeconomic variables.