Proposes a new method for kernel density estimation using stagewise minimization and a simple dictionary.
problem Kernel density estimation with data-adaptive weighting parameters and sparse representation.
method Stagewise minimization algorithm based on U-divergence and a simple dictionary. result Develops non-asymptotic error bound for the proposed estimator.
Forward stagewise regression follows a very simple strategy for constructing a sequence of sparse regression estimates: it starts with all coefficients equal to zero, and iteratively updates the coefficient (by a small amount ε) of the variable that achieves the maximal absolute inner product with the current residua…
Stagewise boosting improves gradient boosting for distributional regression.
problem Vanishing gradient in gradient boosting for distributional regression leads to suboptimal models.
method Proposes a stagewise boosting-type algorithm for distributional regression, combining stagewise regression ideas with gradient boosting and incorporating a novel regularization method, correlation filtering.
result The proposed algorithm provides better results, especially for complex distributions, by reducing the risk of being trapped in a local optimum.
Stagewise training outperforms vanilla SGD in accelerating convergence and testing error reduction.
problem Improving the convergence rate of SGD for neural networks.
method Stagewise training strategy with a geometrically decreasing step size, compared to vanilla SGD with a polynomially decaying step size.
result Stagewise training achieves faster convergence and testing error reduction compared to vanilla SGD under the Polyak-Łojasiewicz condition.
Although stochastic gradient descent (SGD) method and its variants (e.g., stochastic momentum methods, AdaGrad) are the choice of algorithms for solving non-convex problems (especially deep learning), there still remain big gaps between the theory and the practice with many questions unresolved. For example, there is s…
Proposes a model to relate a tensor feature to a univariate outcome using sparse and low-rank components.
problem Relating a univariate outcome to a feature tensor with sparse and low-rank components.
method Divide-and-conquer strategy, stagewise estimation procedure for unit-rank tensor regression.
result The stagewise solution paths converge to those of regularized regression as step size goes to zero.
This paper presents a natural extension of stagewise ranking to the the case of infinitely many items. We introduce the infinite generalized Mallows model (IGM), describe its properties and give procedures to estimate it from data. For estimation of multimodal distributions we introduce the Exponential-Blurring-Mean-Sh…
The performance of EM in learning mixtures of product distributions often depends on the initialization. This can be problematic in crowdsourcing and other applications, e.g. when a small number of 'experts' are diluted by a large number of noisy, unreliable participants. We develop a new EM algorithm that is driven by…
Proposes MM-DUST for efficient generalized lasso solution paths.
problem Efficiently solve generalized lasso problems in large-scale and non-linear models.
method Majorization-minimization dual stagewise algorithm incorporating quadratic majorizers and stagewise learning.
result Established the uniform convergence of approximated solution paths.
We present a method of variable selection for the sparse generalized additive model. The method doesn't assume any specific functional form, and can select from a large number of candidates. It takes the form of incremental forward stagewise regression. Given no functional form is assumed, we devised an approach termed…
Divide-and-conquer method speeds sparse factorization for large matrices.
problem Sparse factorization of large matrices for statistical learning.
method Statistical problem formulation, divide-and-conquer approach, stagewise learning.
result Efficient algorithm with lower complexity than existing methods.
Boosting methods are highly popular and effective supervised learning methods which combine weak learners into a single accurate model with good statistical performance. In this paper, we analyze two well-known boosting methods, AdaBoost and Incremental Forward Stagewise Regression (FSε), by establishing t…
A machine learning approach to record fusion with high accuracy.
problem Aggregating multiple records corresponding to the same entity.
method Constructing feature vectors from attribute-level, record-level, and database-level signals; using a stagewise additive model to learn a classifier.
result Average precision of ~98% with source information and ~94% without source information across diverse datasets.
Least Angle Regression is a promising technique for variable selection applications, offering a nice alternative to stepwise regression. It provides an explanation for the similar behavior of LASSO (ℓ1-penalized regression) and forward stagewise regression, and provides a fast implementation of both. The idea has…
CASP selects reliable policies for two-stage recommender systems by considering both value and support.
problem The selection of a generator in two-stage recommender systems affects both the policy value and the data support used to estimate it.
method CASP combines doubly robust value estimation with a support-burden penalty.
result CASP selects lower-burden policies when estimated value and support credibility are in tension.
Paper analyzes convergence of memory-based distributed SGD with momentum.
problem High communication cost in distributed SGD for large models.
method Introduces transformation equation to analyze convergence of M-DSGD with momentum.
result Universal convergence analysis for M-DSGD with momentum for convex and non-convex problems.
A new method SEBS optimizes SGD batch size for better performance.
problem Optimizing batch size for SGD to balance training speed and generalization.
method SEBS method uses a multi-stage geometric batch size enlargement scheme.
result SEBS reduces parameter updates without increasing generalization error.
Estimates density ratio for two-sample comparison using tree models.
problem Comparing two distributions given i.i.d. observations.
method Additive tree models with balancing loss for density ratio estimation.
result Bayesian inference provides uncertainty quantification for density ratio.
This paper addresses the general problem of modelling and learning rank data with ties. We propose a probabilistic generative model, that models the process as permutations over partitions. This results in super-exponential combinatorial state space with unknown numbers of partitions and unknown ordering among them. We…
New method selects variables for GP regression using sparse projection.
problem Identifying environmental factors affecting metal corrosion.
method Sparse projection of input variables, gradient descent optimization, non-convex marginal likelihood.
result Proposed method outperforms benchmarks in variable selection accuracy.
This work provides simple algorithms for multi-class (and multi-label) prediction in settings where both the number of examples n and the data dimension d are relatively large. These robust and parameter free algorithms are essentially iterative least-squares updates and very versatile both in theory and in practice. O…
REGAIN learns optimal auxiliary directions for forecast reconciliation.
problem Forecast reconciliation from fixed systems; identifying useful auxiliary directions.
method REGAIN learns normalized auxiliary directions, forecasts induced series, and selects directions by loss reduction.
result Gain-selected auxiliary directions improve forecast quality, especially for residual uncertainty.
Paper tackles robust optimization under uncertainty using nested distance.
problem Optimizing under distributionally robust uncertainty with nested distance.
method Equivalent recursive and dynamic programming reformulations for tractable optimization.
result Optimal robust policies can be found efficiently using convex optimization.
The paper decomposes probabilistic scores into reliability, uncertainty, and information loss.
problem Understanding the reliability and uncertainty of probabilistic predictions.
method Developed decomposition identities for proper losses, quantifying reliability, residual uncertainty, and information gain.
result A three-term identity for classification scores, revealing miscalibration, grouping term, and feature-level uncertainty.
Algorithm infers sampling distribution from i.i.d. samples without supervision.
problem Learning probability distributions from unlabeled data.
method Unsupervised tree boosting using additive tree ensembles and new distributional operations.
result Algorithm outperforms deep learning in multivariate density estimation.
This study improves lookahead Bayesian optimization using rollout approximation.
problem Error propagation in lookahead Bayesian optimization due to model mis-specification.
method Proves rollout's improving nature in lookahead BO and provides guidelines for choosing the rolling horizon.
result Empirical results show rollout improves over myopic and non-myopic BO algorithms.
TSL learns separable models to avoid signal cancellation and off-support extrapolation.
problem Signal cancellation and off-support extrapolation in additive models.
method Tensor Separation Learning (TSL) via stagewise greedy procedure with orthogonal refitting.
result TSL avoids information loss caused by marginalizing higher-order interactions.
Enforcing safety is a key aspect of many problems pertaining to sequential decision making under uncertainty, which require the decisions made at every step to be both informative of the optimal decision and also safe. For example, we value both efficacy and comfort in medical therapy, and efficiency and safety in robo…
Efficiently performs robust and sparse kernel regression.
problem Robust and sparse kernel regression.
method Sign gradient descent and early stopping.
result Sign gradient descent achieves robust and sparse kernel regression efficiently.
In this paper we analyze boosting algorithms in linear regression from a new perspective: that of modern first-order methods in convex optimization. We show that classic boosting algorithms in linear regression, namely the incremental forward stagewise algorithm (FSε) and least squares boosting (LS-Boost($…
STL-SGD accelerates Local SGD by gradually increasing communication periods.
problem Reducing communication complexity in distributed SGD.
method Gradually increasing communication periods while decreasing learning rate.
result Significant reduction in communication complexity with improved convergence rates.
This paper analyzes l1-regularized PageRank for local graph clustering, proving its effectiveness and efficiency.
problem Local graph clustering in large graphs, focusing on recovering a single target cluster given a seed node.
method Statistical analysis of l1-regularized PageRank method for recovery of a target cluster.
result l1-regularized PageRank recovers the full target cluster with bounded false positives and exactly the target cluster if the seed is connected solely to it.
New estimators outperform maximum likelihood without hyper-parameter estimation.
problem Improving system identification performance without hyper-parameter estimation.
method Developed generalized Bayes and closed-form biased estimators using excess MSE.
result New estimators have comparable performance to empirical-Bayes-based regularized estimator.
Dual Bayesian Affine Estimators for Wiener-type state-space models
problem Estimating parameters in Wiener-type state-space models
method Fixed-point architecture combining two affine estimators
result Dual basis-parameter estimator achieves comparable parameter MSE to purely affine estimator
New estimator reduces kernel mean estimation error.
problem Kernel mean estimation in reproducing kernel Hilbert spaces.
method Corrupt data with known distributions and estimate kernel mean under the corrupted distribution.
result The marginalized kernel mean estimator achieves lower estimation error.
Enhances gradient estimates for Hermitian Monge-Ampère equations.
problem Improving estimates for Hermitian Monge-Ampère equations.
method Improves gradient estimates using Evans-Krylov and third derivatives estimates.
result Enhanced estimates for second and third order derivatives.
Paper proposes robust estimators for GANs under Wasserstein contamination.
problem Robust estimation of distributions under contamination.
method Wasserstein GAN-based estimators for location, covariance, and regression.
result Proposed estimators are minimax optimal in many scenarios.
New framework converts offline to online estimation using black-box offline estimators.
problem Convert offline estimation algorithms to online estimation algorithms.
method Oracle-Efficient Online Estimation (OEOE) framework.
result Achieves near-optimal online estimation error via black-box offline estimators.
Proposes variational autoencoder for efficient MMSE estimation.
problem Efficient parameterized MMSE estimation for noisy observations.
method Variational autoencoder models data distribution, approximates MMSE.
result Proposed estimator performs well compared to state-of-the-art.
Paper improves Fisher information estimation methods.
problem Estimating Fisher information for location parameters.
method Revisits and improves Bhattacharya estimator, introduces clipped estimator.
result Clipped estimator shows superior convergence rates in Gaussian noise.
Proposes a robust estimator for RD designs.
problem Estimating treatment effects in RD designs.
method Doubly robust estimator combining two estimators.
result Enhances robustness of treatment effect estimators.
New estimator reduces variance in discrete random variables.
problem Estimating gradients for discrete random variables with reduced variance.
method Sampling without replacement and Rao-Blackwellization.
result Our estimator is the most consistent gradient estimator across different entropy settings.
SCOPE estimator improves covariance and precision matrix estimation.
problem Estimating covariance and precision matrices accurately.
method Distributionally robust optimization with convex spectral divergence.
result SCOPE estimator reduces spectral bias and improves condition number.
We present a multi-task learning approach to jointly estimate the means of multiple independent data sets. The proposed multi-task averaging (MTA) algorithm results in a convex combination of the single-task maximum likelihood estimates. We derive the optimal minimum risk estimator and the minimax estimator, and show t…
Obtaining more accurate equity value estimates is the starting point for stock selection, value-based indexing in a noisy market, and beating benchmark indices through tactical style rotation. Unfortunately, discounted cash flow, method of comparables, and fundamental analysis typically yield discrepant valuation estim…
We find an unbiased estimator for MMD variance.
problem Efficiently estimating the variance of MMD estimators.
method Extending and correcting previous work, we derive an unbiased estimator for MMD variance.
result We provide a truly unbiased estimator for MMD variance with no additional computational cost.
Stochastic volatility modelling of financial processes has become increasingly popular. The proposed models usually contain a stationary volatility process. We will motivate and review several nonparametric methods for estimation of the density of the volatility process. Both models based on discretely sampled continuo…
A new copula estimation method using classification.
problem Estimating copula density from joint and marginal distributions.
method Train a classifier to distinguish joint density from product of marginals.
result Empirically outperforms existing copula estimators.