Forward stagewise regression follows a very simple strategy for constructing a sequence of sparse regression estimates: it starts with all coefficients equal to zero, and iteratively updates the coefficient (by a small amount ) of the variable that achieves the maximal absolute inner product with the current residua…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “stagewise learning”
Stagewise training strategy is widely used for learning neural networks, which runs a stochastic algorithm (e.g., SGD) starting with a relatively large step size (aka learning rate) and geometrically decreasing the step size after a number of iterations. It has been observed that the stagewise SGD has much faster conve…
Although stochastic gradient descent (SGD) method and its variants (e.g., stochastic momentum methods, AdaGrad) are the choice of algorithms for solving non-convex problems (especially deep learning), there still remain big gaps between the theory and the practice with many questions unresolved. For example, there is s…
Proposes a new method for kernel density estimation using stagewise minimization and a simple dictionary.
problem Kernel density estimation with data-adaptive weighting parameters and sparse representation.
method Stagewise minimization algorithm based on -divergence and a simple dictionary.
result Develops non-asymptotic error bound for the proposed estimator.
The performance of EM in learning mixtures of product distributions often depends on the initialization. This can be problematic in crowdsourcing and other applications, e.g. when a small number of 'experts' are diluted by a large number of noisy, unreliable participants. We develop a new EM algorithm that is driven by…
Stagewise boosting improves gradient boosting for distributional regression.
problem Vanishing gradient in gradient boosting for distributional regression leads to suboptimal models.
method Proposes a stagewise boosting-type algorithm for distributional regression, combining stagewise regression ideas with gradient boosting and incorporating a novel regularization method, correlation filtering.
result The proposed algorithm provides better results, especially for complex distributions, by reducing the risk of being trapped in a local optimum.
Proposes MM-DUST for efficient generalized lasso solution paths.
problem Efficiently solve generalized lasso problems in large-scale and non-linear models.
method Majorization-minimization dual stagewise algorithm incorporating quadratic majorizers and stagewise learning.
result Established the uniform convergence of approximated solution paths.
We present a method of variable selection for the sparse generalized additive model. The method doesn't assume any specific functional form, and can select from a large number of candidates. It takes the form of incremental forward stagewise regression. Given no functional form is assumed, we devised an approach termed…
Boosting methods are highly popular and effective supervised learning methods which combine weak learners into a single accurate model with good statistical performance. In this paper, we analyze two well-known boosting methods, AdaBoost and Incremental Forward Stagewise Regression (FS), by establishing t…
A machine learning approach to record fusion with high accuracy.
problem Aggregating multiple records corresponding to the same entity.
method Constructing feature vectors from attribute-level, record-level, and database-level signals; using a stagewise additive model to learn a classifier.
result Average precision of ~98% with source information and ~94% without source information across diverse datasets.
Divide-and-conquer method speeds sparse factorization for large matrices.
problem Sparse factorization of large matrices for statistical learning.
method Statistical problem formulation, divide-and-conquer approach, stagewise learning.
result Efficient algorithm with lower complexity than existing methods.
Distributed stochastic gradient descent~(DSGD) has been widely used for optimizing large-scale machine learning models, including both convex and non-convex models. With the rapid growth of model size, huge communication cost has been the bottleneck of traditional DSGD. Recently, many communication compression methods …
We propose a sparse and low-rank tensor regression model to relate a univariate outcome to a feature tensor, in which each unit-rank tensor from the CP decomposition of the coefficient tensor is assumed to be sparse. This structure is both parsimonious and highly interpretable, as it implies that the outcome is related…
Existing research shows that the batch size can seriously affect the performance of stochastic gradient descent~(SGD) based learning, including training speed and generalization ability. A larger batch size typically results in less parameter updates. In distributed training, a larger batch size also results in less fr…
This paper presents a natural extension of stagewise ranking to the the case of infinitely many items. We introduce the infinite generalized Mallows model (IGM), describe its properties and give procedures to estimate it from data. For estimation of multimodal distributions we introduce the Exponential-Blurring-Mean-Sh…
This paper addresses the general problem of modelling and learning rank data with ties. We propose a probabilistic generative model, that models the process as permutations over partitions. This results in super-exponential combinatorial state space with unknown numbers of partitions and unknown ordering among them. We…
Least Angle Regression is a promising technique for variable selection applications, offering a nice alternative to stepwise regression. It provides an explanation for the similar behavior of LASSO (-penalized regression) and forward stagewise regression, and provides a fast implementation of both. The idea has…
TSL learns separable models to avoid signal cancellation and off-support extrapolation.
problem Signal cancellation and off-support extrapolation in additive models.
method Tensor Separation Learning (TSL) via stagewise greedy procedure with orthogonal refitting.
result TSL avoids information loss caused by marginalizing higher-order interactions.
This work provides simple algorithms for multi-class (and multi-label) prediction in settings where both the number of examples n and the data dimension d are relatively large. These robust and parameter free algorithms are essentially iterative least-squares updates and very versatile both in theory and in practice. O…
REGAIN learns optimal auxiliary directions for forecast reconciliation.
problem Forecast reconciliation from fixed systems; identifying useful auxiliary directions.
method REGAIN learns normalized auxiliary directions, forecasts induced series, and selects directions by loss reduction.
result Gain-selected auxiliary directions improve forecast quality, especially for residual uncertainty.
Paper tackles robust optimization under uncertainty using nested distance.
problem Optimizing under distributionally robust uncertainty with nested distance.
method Equivalent recursive and dynamic programming reformulations for tractable optimization.
result Optimal robust policies can be found efficiently using convex optimization.
The paper decomposes probabilistic scores into reliability, uncertainty, and information loss.
problem Understanding the reliability and uncertainty of probabilistic predictions.
method Developed decomposition identities for proper losses, quantifying reliability, residual uncertainty, and information gain.
result A three-term identity for classification scores, revealing miscalibration, grouping term, and feature-level uncertainty.
Lookahead, also known as non-myopic, Bayesian optimization (BO) aims to find optimal sampling policies through solving a dynamic program (DP) that maximizes a long-term reward over a rolling horizon. Though promising, lookahead BO faces the risk of error propagation through its increased dependence on a possibly mis-sp…
Algorithm infers sampling distribution from i.i.d. samples without supervision.
problem Learning probability distributions from unlabeled data.
method Unsupervised tree boosting using additive tree ensembles and new distributional operations.
result Algorithm outperforms deep learning in multivariate density estimation.
A New Perspective on Boosting in Linear Regression via Subgradient Optimization and Relativesmath.ST
In this paper we analyze boosting algorithms in linear regression from a new perspective: that of modern first-order methods in convex optimization. We show that classic boosting algorithms in linear regression, namely the incremental forward stagewise algorithm (FS) and least squares boosting (LS-Boost($…
Enforcing safety is a key aspect of many problems pertaining to sequential decision making under uncertainty, which require the decisions made at every step to be both informative of the optimal decision and also safe. For example, we value both efficacy and comfort in medical therapy, and efficiency and safety in robo…
Estimates density ratio for two-sample comparison using tree models.
problem Comparing two distributions given i.i.d. observations.
method Additive tree models with balancing loss for density ratio estimation.
result Bayesian inference provides uncertainty quantification for density ratio.
STL-SGD accelerates Local SGD by gradually increasing communication periods.
problem Reducing communication complexity in distributed SGD.
method Gradually increasing communication periods while decreasing learning rate.
result Significant reduction in communication complexity with improved convergence rates.
CASP selects reliable policies for two-stage recommender systems by considering both value and support.
problem The selection of a generator in two-stage recommender systems affects both the policy value and the data support used to estimate it.
method CASP combines doubly robust value estimation with a support-burden penalty.
result CASP selects lower-burden policies when estimated value and support credibility are in tension.
Efficiently performs robust and sparse kernel regression.
problem Robust and sparse kernel regression.
method Sign gradient descent and early stopping.
result Sign gradient descent achieves robust and sparse kernel regression efficiently.
New method selects variables for GP regression using sparse projection.
problem Identifying environmental factors affecting metal corrosion.
method Sparse projection of input variables, gradient descent optimization, non-convex marginal likelihood.
result Proposed method outperforms benchmarks in variable selection accuracy.
Local graph clustering methods aim to find small clusters in very large graphs. These methods take as input a graph and a seed node, and they return as output a good cluster in a running time that depends on the size of the output cluster but that is independent of the size of the input graph. In this paper, we adopt a…
Meta-learning adapts models for unseen tasks across AI, robotics, and NLP.
problem Adapting models to unseen tasks efficiently and accurately.
method Black-box, metric-based, layered, and Bayesian approaches.
result Meta-learning enhances model generalization and adaptation to unseen tasks.
Meta-learning improves neural networks by adapting learning algorithms.
problem Conventional AI approaches solve tasks from scratch, but meta-learning aims to improve the learning algorithm.
method Meta-learning adapts a learning algorithm based on multiple learning episodes.
result Meta-learning can tackle deep learning challenges like data and computation bottlenecks.
Survey explores how transfer learning improves deep reinforcement learning.
problem Challenges in reinforcement learning efficiency and effectiveness.
method Categorizes and analyzes transfer learning approaches.
result Transfer learning enhances reinforcement learning performance.
Study Whittle index learning algorithms for restless bandits with constant stepsizes.
problem Optimizing decisions in restless multi-armed bandits with constant stepsizes.
method Developed Q-learning algorithms with constant stepsizes for index learning in restless bandits, extending to DQN and function approximations.
result The algorithms learn the Whittle index effectively.
Poisson learning doesn't solve graph semi-supervised learning issues.
problem Global information loss in graph-based semi-supervised learning.
method Poisson learning is Laplace regularization with thresholding.
result Poisson learning cannot overcome the global information loss problem.
New unsupervised learning technique learns independent kernels for better machine learning tasks.
problem Improving unsupervised representation learning for machine learning tasks.
method Stacking convolutional transforms using alternating proximal minimization scheme.
result DCTL outperforms shallow version CTL on benchmark datasets.
Meta-learning helps models learn quickly from few samples.
problem Deep learning requires many samples, which are hard to get.
method Meta-learning optimizes models to adapt quickly to new tasks.
result Meta-learning can improve model efficiency and adaptability.
New self-imitation learning method improves performance in continuous control tasks.
problem Improving off-policy learning in continuous control tasks.
method Proposes a n-step lower bound to generalize lower-bound Q-learning and introduces a new family of self-imitation learning algorithms.
result n-step lower bound Q-learning achieves a better trade-off between bias and contraction rate, leading to improved performance.
Deep reinforcement learning finds optimal learning policies for adaptive systems.
problem Finding individualized learning plans for learners with unknown latent traits.
method Formulated as a Markov decision process, applied deep Q-learning with a transition model estimator.
result The algorithm efficiently discovers optimal learning policies with small data sets.
Unified framework explains all types of learning, including brain.
problem Lack of clear explanation for deep learning success.
method Constructing a learning principle that equates all learning to probability estimation.
result Unified understanding of learning across different fields.
Cyclical learning rates improve DRL performance without manual tuning.
problem Manual hyperparameter tuning in DRL is time-consuming and error-prone.
method Proposes cyclical learning rates for DRL problems.
result Cyclical learning achieves similar or better results than fixed learning rates.
PAC Bounds for Imitation and Model-based Batch Learning of Contextual Markov Decision Processescs.LG
Study batch reinforcement learning methods for personalized medical treatments.
problem Batch reinforcement learning for personalized medical treatments.
method Direct policy learning and model-based learning approaches.
result Model-based learning is impossible with finite model classes but feasible with relaxed conditions.
A new meta-meta classification method tackles few-shot learning tasks.
problem Learning with limited data in small-data settings.
method Designing an ensemble of learners for a large set of problems, then learning how to combine them for a new problem.
result Meta-meta classification outperforms traditional meta-learning and ensembling approaches in one-shot learning tasks.
The paper proposes a learning algorithm that improves adaptability and generalization.
problem Improving adaptability and generalization in learning models.
method Learning to meta-learn by meta-finetuning on related tasks before adapting to specific tasks.
result Learning to meta-learn improves adaptability and generalization across various tasks.
A common strategy in modern learning systems is to learn a representation that is useful for many tasks, a.k.a. representation learning. We study this strategy in the imitation learning setting for Markov decision processes (MDPs) where multiple experts' trajectories are available. We formulate representation learning …
This paper surveys meta-learning, online, and continual learning.
problem Combining and understanding meta-learning, online, and continual learning.
method Organizing various problem settings using consistent terminology and formal descriptions.
result Fosters further advancements in meta-learning, online, and continual learning.