The Schatten quasi-norm can be used to bridge the gap between the nuclear norm and rank function, and is the tighter approximation to matrix rank. However, most existing Schatten quasi-norm minimization (SQNM) algorithms, as well as for nuclear norm minimization, are too slow or even impractical for large-scale problem…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “factored norms”
This paper develops a new class of nonconvex regularizers for low-rank matrix recovery. Many regularizers are motivated as convex relaxations of the matrix rank function. Our new factor group-sparse regularizers are motivated as a relaxation of the number of nonzero columns in a factorization of the matrix. These nonco…
New proof shows norms can't explain deep learning's implicit regularization.
problem Understanding the implicit regularization in deep learning.
method Mathematical proof on matrix factorization problems.
result Implicit regularization drives norms towards infinity, suggesting rank minimization is key.
Paper tackles low-rank matrix recovery with column -norm regularization.
problem Low-rank matrix recovery problems with column sparsity constraints.
method Developed alternating majorization-minimization (AMM) methods with extrapolation and hybrid AMM.
result Global convergence analysis and superior performance in matrix completion problems.
factor analysismath.OC
New method for factor analysis using nuclear and norms.
problem Finding a low-rank plus sparse decomposition from noisy covariance matrix.
method Formulated an optimization problem with nuclear norm, norm, and KL divergence. Used alternating minimization algorithm.
result Algorithm effectively decomposes covariance matrices in synthetic and real datasets.
This work shows how penalising bias terms in norm regularisation leads to sparse solutions.
problem Understanding the relation between parameter norm regularization and the sparsity of neural network solutions.
method Analyzes one hidden ReLU layer networks with unidimensional data, showing the norm required for function representation and the importance of the bias term's norm.
result Penalising the bias terms in regularisation leads to sparse solutions, enforcing the uniqueness and sparsity of the minimal norm interpolator.
This paper studies optimal approximation factors in misspecified off-policy RL, identifying key factors under various settings.
problem Understanding optimal approximation factors in misspecified off-policy value function estimation.
method Examined various settings including weighted -norm, norm, state aliasing, and state coverage.
result Established optimal asymptotic approximation factors for different norms and identified two instance-dependent factors for norm.
The Schatten- norm () has been widely used to replace the nuclear norm for better approximating the rank function. However, existing methods are either 1) not scalable for large scale problems due to relying on singular value decomposition (SVD) in every iteration, or 2) specific to some values, e.g., $1/…
Based on a new atomic norm, we propose a new convex formulation for sparse matrix factorization problems in which the number of nonzero elements of the factors is assumed fixed and known. The formulation counts sparse PCA with multiple factors, subspace clustering and low-rank sparse bilinear regression as potential ap…
This paper is concerned with the squared F(robenius)-norm regularized factorization form for noisy low-rank matrix recovery problems. Under a suitable assumption on the restricted condition number of the Hessian for the loss function, we derive an error bound to the true matrix for the non-strict critical points with r…
Paper finds exact Hessian sharpness in deep matrix factorization.
problem Understanding the geometry of loss landscapes in deep matrix factorization.
method Presented the first exact expression for Hessian maximum eigenvalue.
result Spectral-norm balance is a sufficient condition for flatness in deep matrix factorization.
In this note, we derive concentration inequalities for random vectors with subGaussian norm (a generalization of both subGaussian random vectors and norm bounded random vectors), which are tight up to logarithmic factors.
New ONMF model minimizes KL divergence for better sparse data modeling.
problem Clustering and data modeling with sparse vectors.
method Developed KL-ONMF algorithm based on alternating optimization.
result KL-ONMF outperforms Frobenius-norm ONMF for document classification and hyperspectral image unmixing.
Study shows interpolating predictor's risk is optimal in low-dimensional factor regression models.
problem Understanding the risk of interpolating predictors in high-dimensional factor regression models.
method Detailed finite-sample analysis of minimum-norm interpolating predictor's risk in factor regression models.
result The risk of the minimum-norm interpolating predictor approaches optimal benchmarks in low-dimensional factor regression models.
KL property of exponent of -norm and DC regularized factorizations for low-rank matrix recoverymath.OC
This paper is concerned with the factorization form of the rank regularized loss minimization problem. To cater for the scenario in which only a coarse estimation is available for the rank of the true matrix, an -norm regularized term is added to the factored loss function to reduce the rank adaptively; and…
Unified framework for estimating high-dimensional conditional factor models.
problem Estimating high-dimensional conditional latent factor models with practical limitations.
method Constrained nuclear norm regularization and cross-validation for parameter selection.
result Imposing homogeneity improves model predictability, with new method outperforming alternatives.
DoRA improves adaptation efficiency for large models by factoring norms and fusing kernels.
problem High-rank DoRA is computationally expensive and infeasible on common GPUs.
method Factored norms and fused Triton kernels to reduce memory and speed up computation.
result Fused implementation is up to 2.0x faster for inference and 1.9x faster for gradient computation.
We make an estimation of the value of the Gromov norm of the Cartesian product of two surfaces. Our method uses a connection between these norms and the minimal size of triangulations of the products of two polygons. This allows us to prove that the Gromov norm of this product is between 32 and 52 when both factors hav…
We study the projected gradient descent method on low-rank matrix problems with a strongly convex objective. We use the Burer-Monteiro factorization approach to implicitly enforce low-rankness; such factorization introduces non-convexity in the objective. We focus on constraint sets that include both positive semi-defi…
New algorithms improve approximation of matrix norms, with applications in statistics and machine learning.
problem Improving approximation of matrix norms for in polynomial time.
method Polynomial-time multiplicative approximation algorithms for norm, leveraging sum-of-squares certificates.
result Achieved polynomially improved approximation factors, notably for .
A new model reduces noise and speeds up subspace segmentation.
problem Subspace segmentation from noisy data.
method Group norm regularized factorization model (GNRFM) with AALM algorithm.
result The method is faster and more robust to noise.
New model reduces matrix factorization bias, yielding truly low-rank solutions.
problem Gradient descent's implicit bias in matrix factorization.
method Introducing a new factorization model with constrained factors and diagonal components.
result The new model consistently exhibits a strong implicit bias, yielding truly low-rank solutions.
Gradient flow with infinitesimal initialization converges to Greedy Low-Rank Learning for matrix factorization.
problem Understanding implicit regularization in gradient descent for matrix factorization.
method Theoretical and empirical analysis of gradient flow with infinitesimal initialization and Greedy Low-Rank Learning.
result Gradient flow with infinitesimal initialization is mathematically equivalent to Greedy Low-Rank Learning for depth-2 matrix factorization under reasonable assumptions.
The heavy-tailed distributions of corrupted outliers and singular values of all channels in low-level vision have proven effective priors for many applications such as background modeling, photometric stereo and image alignment. And they can be well modeled by a hyper-Laplacian. However, the use of such distributions g…
We study implicit regularization when optimizing an underdetermined quadratic objective over a matrix with gradient descent on a factorization of . We conjecture and provide empirical and theoretical evidence that with small enough step sizes and initialization close enough to the origin, gradient descent on a f…
Regularization for matrix factorization (MF) and approximation problems has been carried out in many different ways. Due to its popularity in deep learning, dropout has been applied also for this class of problems. Despite its solid empirical performance, the theoretical properties of dropout as a regularizer remain qu…
Dropout is a simple yet effective algorithm for regularizing neural networks by randomly dropping out units through Bernoulli multiplicative noise, and for some restricted problem classes, such as linear or logistic regression, several theoretical studies have demonstrated the equivalence between dropout and a fully de…
Improved sample complexity for ReLU networks with norm constraints.
problem Estimating sample complexity for ReLU networks under norm constraints.
method Refined Rademacher complexity analysis for function class.
result Often no explicit depth-dependence in sample complexity bound.
Paper analyzes convergence of PAM method for low-rank factorization models.
problem Convergence analysis of PAM method with subspace correction for low-rank factorization models.
method Majorized proximal alternating minimization (PAM) method with subspace correction.
result Established full convergence of PAM method under KL property and column -norm condition.
We model how Lipschitz continuity changes during neural network training.
problem Understanding how Lipschitz continuity evolves during training.
method We use a system of stochastic differential equations to capture the dynamics of Lipschitz continuity under SGD.
result We identify three factors driving the evolution of Lipschitz continuity: gradient flow projection, gradient noise, and Hessian projection.
Recovering low-rank and sparse matrices from incomplete or corrupted observations is an important problem in machine learning, statistics, bioinformatics, computer vision, as well as signal and image processing. In theory, this problem can be solved by the natural convex joint/mixed relaxations (i.e., l_{1}-norm and tr…
This paper tackles fitting multilevel low rank matrices by addressing three problems.
problem Fitting a given matrix by an MLR matrix in the Frobenius norm.
method Factor fitting, rank allocation, and hierarchical partitioning.
result The proposed methods can fit a given matrix by an MLR matrix in the Frobenius norm.
In this note we determine all possible dominations between different products of manifolds, when none of the factors of the codomain is dominated by products. As a consequence, we determine the finiteness of every product-associated functorial semi-norm on the fundamental classes of the aforementioned products. These r…
New bounds improve deep learning performance efficiently.
problem Improving generalization and robustness of deep learning models.
method Deriving four provable upper bounds on spectral norm of convolution layers, differentiable and efficient.
result Minimum of four bounds is a tight, differentiable and efficient upper bound on spectral norm.
We propose an algorithm for the non-negative factorization of an occurrence tensor built from heterogeneous networks. We use l0 norm to model sparse errors over discrete values (occurrences), and use decomposed factors to model the embedded groups of nodes. An efficient splitting method is developed to optimize the non…
The Schatten quasi-norm was introduced to bridge the gap between the trace norm and rank function. However, existing algorithms are too slow or even impractical for large-scale problems. Motivated by the equivalence relation between the trace norm and its bilinear spectral penalty, we define two tractable Schatten norm…
New bounds adaptively control spectral complexity of trained Transformers.
problem Understanding why Transformers generalize well in machine learning.
method Spectrum-adaptive post hoc generalization bounds for multi-layer Transformers.
result Bounds adaptively trade off spectral complexity against dimension and depth factors.
On the Loss Landscape Geometry of Regularized Deep Matrix Factorization: Uniqueness and Sharpnessstat.ML
The paper studies the loss landscape of regularized deep matrix factorization, revealing unique and sharp minimizers.
problem Understanding the loss landscape and minimizers of regularized deep matrix factorization problems.
method Theoretical analysis of -regularized deep matrix factorization/deep linear network training problems with squared-error loss.
result The unique end-to-end minimizer exists for all target matrices except for a set of Lebesgue measure zero.
Unified framework for coupled tensor completion improves recovery accuracy.
problem Improving recovery accuracy in coupled tensor completion.
method Unified framework using tensor ring (TR) decomposition with shared latent factors and novel optimization model.
result The proposed method achieves superior recovery accuracy on real-world data compared to state-of-the-art methods.
We study the adaptive estimation of copula correlation matrix for the semi-parametric elliptical copula model. In this context, the correlations are connected to Kendall's tau through a sine function transformation. Hence, a natural estimate for is the plug-in estimator with Kendall's tau statistic. We …
The study analyzes robustness of estimators in linear models with adversarial errors.
problem Analyzing robustness of estimators in linear models with adversarial errors.
method Develops a general theory for minimum norm interpolating estimators and RERM in linear models without conditions on errors.
result Quantitative bound for the prediction error relating it to Rademacher complexity, norm of minimum norm interpolator of errors, and subdifferential size.
Max-norm regularizer has been extensively studied in the last decade as it promotes an effective low-rank estimation for the underlying data. However, such max-norm regularized problems are typically formulated and solved in a batch manner, which prevents it from processing big data due to possible memory budget. In th…
Recently, locality sensitive hashing (LSH) was shown to be effective for MIPS and several algorithms including -ALSH, Sign-ALSH and Simple-LSH have been proposed. In this paper, we introduce the norm-range partition technique, which partitions the original dataset into sub-datasets containing items with similar 2-…
Paper shows no spurious local minima in a specific matrix factorization problem.
problem Optimization of -norm rank-one symmetric matrix factorization.
method Second-order variational analysis to study the landscape of the problem.
result Any second-order stationary point is globally optimal.
Gradient descent promotes low-rank solutions in tensor completion.
problem Implicit regularization in tensor factorization using gradient descent.
method Introduced deep Tucker and TensorTrain (TT) unconstrained factorization to address tensor completion.
result Gradient descent promotes solutions with low-rank.
New bounds show current methods overestimate system parameter errors.
problem Current bounds overestimate parameter errors in system identification.
method Utilized asymptotic normality and second-order decomposition.
result Obtained finite-sample bounds matching optimal rates up to constants.
The paper tackles tensor factorization and completion from noisy data.
problem Sparse nonnegative tensor factorization and completion from partial and noisy observations.
method Minimizes the sum of maximum likelihood estimation and tensor norm with nonnegativity constraints.
result Error bounds and minimax lower bounds are established for the proposed model.
The paper studies the metric and algebraic structures on section rings of projective manifolds.
problem Understanding the relationship between metric and algebraic structures on section rings.
method Analyzes the section ring of projective manifolds and ample line bundles, proving approximate isometry properties under various norms.
result Characterizes -norms associated with continuous plurisubharmonic metrics and refines the theorem of Phong-Sturm.