A new method for selecting variables in nonparametric models without assuming specific forms.
problem Variable selection in sparse generalized additive models without assuming specific functional forms.
method Incremental forward stagewise regression with roughening to adjust residuals.
result The method performs competitively and is demonstrated on real datasets.
Boosting methods are highly popular and effective supervised learning methods which combine weak learners into a single accurate model with good statistical performance. In this paper, we analyze two well-known boosting methods, AdaBoost and Incremental Forward Stagewise Regression (FSε), by establishing t…
Forward stagewise regression follows a very simple strategy for constructing a sequence of sparse regression estimates: it starts with all coefficients equal to zero, and iteratively updates the coefficient (by a small amount ε) of the variable that achieves the maximal absolute inner product with the current residua…
Stagewise boosting improves gradient boosting for distributional regression.
problem Vanishing gradient in gradient boosting for distributional regression leads to suboptimal models.
method Proposes a stagewise boosting-type algorithm for distributional regression, combining stagewise regression ideas with gradient boosting and incorporating a novel regularization method, correlation filtering.
result The proposed algorithm provides better results, especially for complex distributions, by reducing the risk of being trapped in a local optimum.
Least Angle Regression is a promising technique for variable selection applications, offering a nice alternative to stepwise regression. It provides an explanation for the similar behavior of LASSO (ℓ1-penalized regression) and forward stagewise regression, and provides a fast implementation of both. The idea has…
Proposes a new method for kernel density estimation using stagewise minimization and a simple dictionary.
problem Kernel density estimation with data-adaptive weighting parameters and sparse representation.
method Stagewise minimization algorithm based on U-divergence and a simple dictionary. result Develops non-asymptotic error bound for the proposed estimator.
This paper addresses convergence issues in non-convex optimization problems using stagewise learning.
problem Theoretical gaps in convergence for non-convex problems and lack of adaptive step size theories.
method Proposes a stagewise optimization framework for non-smooth non-convex problems using adaptive step sizes.
result Demonstrates adaptive convergence of stagewise AdaGrad and improved generalization performance.
Stagewise training outperforms vanilla SGD in accelerating convergence and testing error reduction.
problem Improving the convergence rate of SGD for neural networks.
method Stagewise training strategy with a geometrically decreasing step size, compared to vanilla SGD with a polynomially decaying step size.
result Stagewise training achieves faster convergence and testing error reduction compared to vanilla SGD under the Polyak-Łojasiewicz condition.
Divide-and-conquer method speeds sparse factorization for large matrices.
problem Sparse factorization of large matrices for statistical learning.
method Statistical problem formulation, divide-and-conquer approach, stagewise learning.
result Efficient algorithm with lower complexity than existing methods.
In this paper we analyze boosting algorithms in linear regression from a new perspective: that of modern first-order methods in convex optimization. We show that classic boosting algorithms in linear regression, namely the incremental forward stagewise algorithm (FSε) and least squares boosting (LS-Boost($…
Paper presents IMRCs for evolving tasks with forward and backward learning.
problem Incremental learning of evolving tasks with few samples per task.
method Incremental minimax risk classifiers (IMRCs) that exploit forward and backward learning.
result IMRCs provide significant performance improvement, especially with reduced sample sizes.
The performance of EM in learning mixtures of product distributions often depends on the initialization. This can be problematic in crowdsourcing and other applications, e.g. when a small number of 'experts' are diluted by a large number of noisy, unreliable participants. We develop a new EM algorithm that is driven by…
Proposes MM-DUST for efficient generalized lasso solution paths.
problem Efficiently solve generalized lasso problems in large-scale and non-linear models.
method Majorization-minimization dual stagewise algorithm incorporating quadratic majorizers and stagewise learning.
result Established the uniform convergence of approximated solution paths.
A new generative classification strategy outperforms existing methods in class-incremental learning.
problem Incrementally training deep neural networks to recognize new classes is challenging.
method Proposes learning the joint distribution p(x,y) and performing classification using Bayes' rule, implemented with variational autoencoders and importance sampling.
result Performs very well on continual learning benchmarks, outperforming existing baselines.
Proposes a model to relate a tensor feature to a univariate outcome using sparse and low-rank components.
problem Relating a univariate outcome to a feature tensor with sparse and low-rank components.
method Divide-and-conquer strategy, stagewise estimation procedure for unit-rank tensor regression.
result The stagewise solution paths converge to those of regularized regression as step size goes to zero.
Study shows accuracy of neural networks depends more on error location than percentage of error.
problem Effect of noise on accuracy in incremental learning neural networks.
method Empirical study using Perceptron, Feed Forward Neural Network, and Radial Basis Function Neural Network.
result Accuracy of neural networks is more dependent on error location than the percentage of error.
A machine learning approach to record fusion with high accuracy.
problem Aggregating multiple records corresponding to the same entity.
method Constructing feature vectors from attribute-level, record-level, and database-level signals; using a stagewise additive model to learn a classifier.
result Average precision of ~98% with source information and ~94% without source information across diverse datasets.
Bayesian inference over admissible histories leads to irreversible kinetics.
problem Modeling irreversible processes in systems with uncertain histories.
method A Gibbs-type measure weighted by energy-dissipation action and observation constraints, interpreted as a Bayesian posterior.
result The measure concentrates on maximum-a-posteriori (MAP) histories, recovering classical deterministic evolution.
Paper presents CeNN quantization for efficient CPS applications.
problem Efficient processing for CPS applications, especially in telemedicine and ADAS.
method Incremental quantization of CeNNs with various strategies.
result Achieved up to 7.8x speedup with no performance loss.
Efficiently performs robust and sparse kernel regression.
problem Robust and sparse kernel regression.
method Sign gradient descent and early stopping.
result Sign gradient descent achieves robust and sparse kernel regression efficiently.
This paper presents a natural extension of stagewise ranking to the the case of infinitely many items. We introduce the infinite generalized Mallows model (IGM), describe its properties and give procedures to estimate it from data. For estimation of multimodal distributions we introduce the Exponential-Blurring-Mean-Sh…
This paper tackles few-shot classification by improving GAN-based data augmentation.
problem Improving few-shot classification performance using GANs with limited data.
method Fine-tuning GANs for few-shot classification, addressing training and evaluation challenges.
result Semi-supervised fine-tuning is a more effective approach for few-shot classification with limited data.
New method selects variables for GP regression using sparse projection.
problem Identifying environmental factors affecting metal corrosion.
method Sparse projection of input variables, gradient descent optimization, non-convex marginal likelihood.
result Proposed method outperforms benchmarks in variable selection accuracy.
Estimates density ratio for two-sample comparison using tree models.
problem Comparing two distributions given i.i.d. observations.
method Additive tree models with balancing loss for density ratio estimation.
result Bayesian inference provides uncertainty quantification for density ratio.
A new QHR model extends HR model with a quadratic variance function.
problem Modeling volatility with greater flexibility and stationarity.
method Introducing a quadratic variance function to the HR model, maintaining Markovian property.
result Stationary distribution of the QHR model is Pearson type IV.
Paper analyzes convergence of memory-based distributed SGD with momentum.
problem High communication cost in distributed SGD for large models.
method Introduces transformation equation to analyze convergence of M-DSGD with momentum.
result Universal convergence analysis for M-DSGD with momentum for convex and non-convex problems.
We introduce incremental variational inference and apply it to latent Dirichlet allocation (LDA). Incremental variational inference is inspired by incremental EM and provides an alternative to stochastic variational inference. Incremental LDA can process massive document collections, does not require to set a learning …
Algorithm infers sampling distribution from i.i.d. samples without supervision.
problem Learning probability distributions from unlabeled data.
method Unsupervised tree boosting using additive tree ensembles and new distributional operations.
result Algorithm outperforms deep learning in multivariate density estimation.
A new method SEBS optimizes SGD batch size for better performance.
problem Optimizing batch size for SGD to balance training speed and generalization.
method SEBS method uses a multi-stage geometric batch size enlargement scheme.
result SEBS reduces parameter updates without increasing generalization error.
Study finds anomalies in high-frequency S&P 500 price changes.
problem Anomalies in high-frequency S&P 500 price changes.
method Using NBBO event-time data, the study forms pairs of backward and forward price increments, standardizes them, and estimates expected responses on a fine grid of push magnitudes.
result Persistent structural shift in expected responses: near zero for short lags, pronounced tails for long lags, indicating correlation between larger historical pushes and nonzero responses.
Efficiently price VIX options using multilevel Monte Carlo in rough Bergomi model.
problem Pricing VIX options in a rough Bergomi model with high computational complexity.
method Combining rectangle discretization, Cholesky sampling, and multilevel Monte Carlo.
result Reduced computational complexity to O(ε−2log2(ε)) and asymptotically optimal O(ε−2). ContinuousNet generalizes ResNets to continuous dynamical systems.
problem ResNets fail to be meaningful dynamical integrators.
method Embedding continuous dynamical systems into higher-order numerical integration schemes (Runge Kutta).
result ContinuousNet exhibits invariance to discrete time step sizes and numerical integration schemes.
A new algorithm for faster incremental kernel PCA and Nyström approximation.
problem Efficiency in handling streaming data and large datasets.
method Incremental kernel PCA and Nyström approximation using rank one updates.
result More computationally and memory efficient than existing methods.
The paper extends cluster validity indices for incremental analysis.
problem Providing incremental alternatives for cluster validation.
method Extending iCVI family to include 6 incremental indices and examining their behavior under under- and over-partitioning.
result Over-partitioning is more challenging to detect than under-partitioning.
EML model tackles evolving features in online metric learning.
problem Challenges in applying metric learning to evolving features.
method Develops a new Evolving Metric Learning (EML) model for incremental and decremental features.
result EML model handles instance and feature evolutions simultaneously.
New approach AR1 improves performance in class-incremental learning.
problem Training deep models sequentially on a single incremental task without forgetting.
method Combining architectural and regularization strategies, AR1 is specifically designed for incremental task scenarios.
result AR1 outperformed existing regularization strategies on CORe50 and iCIFAR-100.
EILearn learns incrementally using previous classifier knowledge.
problem Incremental learning with previous data.
method Retains and uses previous classifier knowledge, monitors performance, eliminates poorly performing classifiers.
result Outperforms existing incremental learning approaches.
Paper proposes faster incremental subclass discriminant analysis.
problem Efficiently classify subclasses in incremental data.
method Exact and approximate linear and kernelized solutions.
result Superior training time and accuracy compared to existing methods.
Depth helps neural networks learn simpler solutions incrementally.
problem Understanding why deep neural networks generalize well despite complex architectures.
method Formal definition of incremental learning dynamics, theoretical analysis of depth and initialization effects, experiments with various models.
result Incremental learning dynamics can arise in deeper models, but not in shallow ones, under specific conditions.
Paper tackles few-shot class-incremental learning with a neural gas network.
problem Incrementally learn new classes from very few labelled samples without forgetting old classes.
method Proposes TOPIC framework using a neural gas network to preserve class topology and adapt to new samples.
result Significantly outperforms other methods on CIFAR100, miniImageNet, and CUB200 datasets.
Study of electronic corn futures trading shows discrete price changes and non-Gaussian distributions.
problem Discrepancy between theoretical continuous price models and actual intra-day trading data.
method Analysis of discrete price increments, volume, and profit strategies using statistical distributions and probability theory.
result Kumaraswamy distribution better fits waiting times than Weibull, and price jumps resemble branching reactions.
iCaRL learns new classes incrementally from data streams.
problem Incremental learning of new concepts over time from a stream of data.
method iCaRL: learns strong classifiers and data representation simultaneously, allowing progressive addition of new classes.
result iCaRL can learn many classes incrementally over a long period of time, unlike other methods.
Paper tackles private incremental regression in streaming data.
problem Private incremental regression in streaming data.
method Generic transformation of private batch ERM to incremental ERM, and two mechanisms for private incremental regression.
result Improved empirical risk for certain regression problems.
Deep Incremental Boosting speeds up Deep Learning training with improved generalization.
problem Reducing training time and improving generalization in Deep Learning.
method Inspired by AdaBoost and Transfer Learning, Deep Incremental Boosting adapts Ensemble methods to Deep Learning.
result Reduces training time and improves generalization on Deep Learning datasets.
Paper introduces a new method for learning robot dynamics.
problem Learning robot dynamics with limited prior information.
method Incremental semiparametric inverse dynamics learning combining parametric and nonparametric models.
result Validated the approach on a humanoid robot arm.
A new model learns preferences incrementally without personal data.
problem Incremental session-based recommendation without personal data.
method Memory Augmented Neural model (MAN) that combines a neural recommender with a nonparametric memory.
result MAN consistently outperforms existing methods in incremental session-based recommendation.
Paper proposes CNE-net to tackle incremental learning in (T)ACSA tasks.
problem Catastrophic forgetting in multi-task incremental learning for (T)ACSA.
method Category Name Embedding network (CNE-net) with shared encoder and decoder.
result State-of-the-art performance on (T)ACSA benchmark datasets.
New method for clustering large multi-view data.
problem Handling large multi-view data efficiently.
method Incremental minimax optimization based fuzzy clustering (IminimaxFCM).
result IminimaxFCM outperforms related methods in clustering accuracy.