New method uses ResNet's feature optimization for gradient boosting.
problem Improving gradient boosting methods for classification tasks.
method Functional Gradient Boosting based on Residual Network Perception.
result ResFGB method outperforms state-of-the-art methods like LightGBM.
An ensemble approach learns vector-weighted formulae for RLR.
problem Learning Relational Logistic Regression (RLR) with vector-weighted features.
method Functional-gradient boosting methods for probabilistic logic models.
result Our approach outperforms other methods for learning RLR.
The paper provides a uniform convergence bound for smooth calibration error and its relationship with functional gradient.
problem Limited theoretical understanding of learning algorithms achieving high accuracy and good calibration.
method Focuses on smooth calibration error, providing a uniform convergence bound and proving the relationship with functional gradient.
result Derives conditions for simultaneous classification and calibration guarantees in gradient boosting trees, kernel boosting, and neural networks.
Proposes statistical inference for L2-Boosting.
problem Statistical inference for L2-Boosting. method Post-selection inference for iterative variable selection in L2-Boosting. result Developed tests and confidence intervals for L2-Boosting. Boosting methods for interval-censored data improve predictive accuracy in survival analysis.
problem Handling interval-censored data in survival analysis and time-to-event studies.
method Nonparametric boosting methods using censoring unbiased transformations and functional gradient descent.
result Effective boosting methods for regression and classification with interval-censored data, offering robust performance.
Boosting method adapted for non-square loss functions.
problem Applying gradient boosting to non-square loss functions.
method Introducing column measure ν^(L_2) to adapt L2-Boosting for different loss functions.
result L2-Boosting can be modified to select variables according to different loss functions.
Gradient boosting method enforced with individual fairness.
problem Enforcing fairness in machine learning models.
method Functional gradient descent on robust loss function.
result Algorithm converges globally and generalizes.
A new gradient tree boosting framework reduces variance and accelerates performance.
problem High variance in stochastic gradient boosting.
method Combining gradient tree boosting with importance sampling and a regularizer.
result Achieves a linear convergence rate on logistic loss and 2.5x--18x acceleration on LogitBoost and LambdaMART.
RFRBoost uses random features to boost deep residual neural networks, improving performance and computational efficiency.
problem Improving performance of deep residual neural networks (RFNNs) while preserving convex optimization benefits.
method Random Feature Representation Boosting (RFRBoost) using boosting theory and random features at each layer.
result RFRBoost significantly outperforms RFNNs and end-to-end trained MLP ResNets in small- to medium-scale tabular datasets.
Given functional data from a survival process with time-dependent covariates, we derive a smooth convex representation for its nonparametric log-likelihood functional and obtain its functional gradient. From this, we devise a generic gradient boosting procedure for estimating the hazard function nonparametrically. An i…
New bounds show complex neural networks need many queries to learn.
problem Learning non-polynomial activation functions with Gaussian marginals.
method Gradient boosting procedure to amplify lower bounds on SQ dimension of neural networks.
result Statistical-query lower bounds for ReLU regression with 2ncε queries. Python package for ordinal regression using gradient boosting.
problem Handling ordinal variables in machine learning.
method Gradient boosting with latent variable framework.
result Performs joint optimization of latent function and threshold vector.
SIFG uses noisy particles to efficiently sample from complex distributions.
problem Efficient sampling from complex distributions using particle-based methods.
method SIFG introduces a semi-implicit functional gradient flow with Gaussian noise to improve sampling efficiency and accuracy.
result SIFG achieves strong theoretical convergence guarantees and efficient sampling.
New method handles infinite models with rigorous L1-constraint.
problem Optimizing infinite-dimensional spaces with L1-regularization. method Stochastic Particle Gradient Descent for transport maps.
result Obtains transport map for infinite ensemble learning.
Adapting functional gradients improves FGD's practicality and theoretical guarantees.
problem Implementing FGD in practice due to infinite-dimensional functional gradients.
method Adapting the representation of functional gradients.
result Establishes convergence to a stationary point for smooth losses and a global minimizer under smoothness + Polyak-Lojasiewicz condition.
Paper proposes a new method for training nonconvex models.
problem Training nonconvex models like neural networks.
method Successive functional gradient optimization using mirror descent in a function space.
result The method leads to better performance than standard training techniques.
A new recursive mixture estimation algorithm improves VAE inference efficiency and accuracy.
problem Inaccurate posterior approximation in traditional VAEs.
method Recursive mixture estimation algorithm using functional gradient approach for iterative component selection.
result Significantly higher test data likelihood compared to state-of-the-art methods on benchmark datasets.
Enhances GAN training by adding a gradient layer to improve convergence.
problem Degeneration of convergence speed and limited representational power in GANs.
method Introduces a gradient layer to seek a descent direction in an infinite-dimensional space, bypassing local optima.
result Demonstrates faster convergence through numerical experiments.
New method improves GAN by reducing KL divergence.
problem Improving stability and effectiveness of GANs.
method Theory-based approach using functional gradient learning.
result KL divergence between real and generated data converges to zero.
New method for constrained sampling using gradient flows.
problem Sampling from constrained domains.
method Introducing a boundary condition for gradient flow to confine particles within the domain.
result Provable continuous-time convergence in total variation for constrained sampling.
A new ParVI framework improves particle-based variational inference methods.
problem Non-trivial kernel design in particle-based variational inference methods.
method Proposes a generalized Wasserstein gradient descent (GWG) framework with broader regularizers.
result Demonstrates strong convergence guarantees and effectiveness on simulated and real data.
Paper improves variance control in importance weighted variational bounds.
problem Improving the variance of gradient estimators for IWAE.
method Develops a novel control variate that grows SNR as √K for large K.
result Empirically, the method yields superior variance reduction for generative models.
New particle-based VI algorithm expands function class and improves scalability.
problem Limited function class in particle-based VI algorithms restricts flexibility and scalability.
method Introduces a functional regularization term to expand the function class and proposes PFG algorithm.
result Proposed PFG algorithm has larger function class, improved scalability, better adaptation to ill-conditioned distributions, and provable convergence.
MetaFun learns functional representations for meta-learning.
problem Few-shot classification on large-scale datasets.
method Functional encoder-decoder approach with iterative updates.
result State-of-the-art performance on miniImageNet and tieredImageNet.
The general perception is that kernel methods are not scalable, and neural nets are the methods of choice for nonlinear learning problems. Or have we simply not tried hard enough for kernel methods? Here we propose an approach that scales up kernel methods using a novel concept called "doubly stochastic functional grad…
New model uses attention for in-context learning of categorical data.
problem Learning from categorical data in context.
method Attention-based network with self-attention and cross-attention layers, using functional gradient descent.
result Model can perform multi-step inference for categorical observations.
Early stopping improves nonparametric testing optimality.
problem Improving minimax optimal testing in nonparametric settings.
method Applying early stopping to functional gradient descent in RKHS to obtain a Wald-type test.
result Sharp stopping rule for optimal testing in nonparametric settings.
Paper solves barycenter of probability distributions using Sinkhorn divergence.
problem Computing the barycenter of a set of probability distributions under the Sinkhorn divergence.
method Recast as unconstrained functional optimization and develop Sinkhorn Descent (SD) method.
result SD converges to a stationary point at a sublinear rate and asymptotically finds a global minimizer.
Generative ParVI learns flexible sampling from posterior distributions.
problem Avoiding arbitrary parametric assumptions in variational inference.
method Neural sampler trained with functional gradient of KL-divergence.
result GPVI outperforms previous generative ParVI methods and is competitive with other approaches.
We develop the calculus for hypersurface variations based on variation of the hypersurface defining function. This is used to show that the functional gradient of a new Willmore-like, conformal hypersurface energy agrees exactly with the obstruction to smoothly solving the singular Yamabe problem for conformally compac…
Gradient of harmonic functions tied to level hypersurface geometry.
problem Understanding how the gradient of harmonic functions changes.
method Analyzes how the gradient of harmonic functions changes along gradient flows.
result The gradient's change is determined by the mean curvature of level hypersurfaces.
The paper estimates gradients of positive pseudo-harmonic functions on complex manifolds.
problem Estimating gradients of positive solutions on pseudo-Hermitian manifolds.
method Using the equation Δ_b u = - λu, the authors derive a gradient estimate.
result The gradient estimate leads to a Liouville theorem for positive pseudo-harmonic functions.
Paper relaxes optimal transport using convex functions for data science.
problem Optimal transport problem on finite spaces.
method Relaxation via strictly convex functions (Kullback-Leibler divergence, Bregman divergences). Gradient descent iterative process.
result Mathematical foundations and iterative process for the relaxed optimal transport problem.
Gradient descent biases towards stable rank networks for nearly-orthogonal data.
problem Understanding implicit bias in non-smooth neural networks trained by gradient descent.
method Analysis of two-layer ReLU and leaky ReLU networks trained by gradient descent on nearly-orthogonal data.
result Gradient descent biases towards networks with stable rank and uniform margin for nearly-orthogonal data.
Proves properties of Morse vector fields on compact manifolds.
problem Properties of gradient vector fields of Morse functions.
method Analyzes connectedness of critical points and shrinkage of flow.
result Shows connectedness of critical points through orbits and exponential shrinkage.
Paper proves constant functions for pluriharmonic on certain solitons.
problem Proving Liouville type theorems for harmonic functions on gradient Ricci solitons.
method Analyzing pluriharmonic functions on gradient shrinking or steady Kähler-Ricci solitons.
result Any pluriharmonic function with gradient in Lp is constant. Paper proves a Liouville theorem for solitons with constant curvature.
problem Understanding harmonic functions on specific geometric structures.
method Proved a Liouville theorem without gradient estimates.
result Finite dimensionality of harmonic functions with polynomial growth.
Deep networks can approximate smooth functions by compositions of nearly identity functions.
problem Optimizing deep networks for smooth function approximation.
method Representing smooth functions as compositions of near-identity functions with decreasing Lipschitz constants.
result Functional gradient methods for residual networks avoid suboptimal critical points in the near-identity region.
Fair MP-Boost improves fairness and interpretability in boosting methods.
problem Improving fairness and interpretability in boosting methods.
method Fair MP-Boost uses adaptive sampling of minipatches to balance accuracy and fairness.
result Fair MP-Boost enhances fairness and accuracy while providing interpretable feature importance.
BooST combines boosting and smooth trees for nonlinear regression.
problem Estimating partial effects in nonlinear models.
method Boosting smooth trees for partial effect estimation.
result BooST provides more interpretable partial effects than tree-based models.
Survey of gradient boosting algorithms and their applications.
problem Improving machine learning models through boosting techniques.
method Detailed mathematical frameworks of gradient boosting algorithms.
result Illustration of gradient boosting's role in model optimization and ranking.
Proximal boosting improves gradient boosting for non-differentiable losses.
problem Minimizing non-differentiable losses in prediction models.
method Proximal point algorithm applied to gradient boosting.
result Proximal boosting outperforms gradient boosting in convergence rate and accuracy.
A scalable framework for gradient boosting using TensorFlow.
problem Training gradient boosted trees efficiently on large datasets.
method Distributed training architecture, automatic loss differentiation, layer-by-layer boosting, multi-class handling, regularization.
result Faster prediction and smaller ensembles compared to traditional methods.
MP-Boost boosts accuracy faster and more interpretable than AdaBoost.
problem Developing a faster, more interpretable boosting method.
method Adaptive selection of minipatches (small subsets of instances and features) at each iteration.
result Achieves comparable accuracy to AdaBoost and gradient boosting but faster and more interpretable.
Boost-R uses gradient boosted trees for analyzing recurrence data.
problem Analyzing recurrence data with static and dynamic features.
method Gradient boosted additive trees with time-dependent functions.
result Estimates the cumulative intensity function of recurrent event processes.
Residual Networks are shown to be equivalent to boosting feature representation.
problem Improving feature representation in deep learning models.
method Proved ResNet's equivalence to Online Gradient Boosting and proposed decision tree residual modules.
result ResNet can achieve Online Gradient Boosting regret bounds through architectural changes.
This research compares gradient and Newton boosting methods in classification and regression.
problem The distinction between gradient descent and Newton updates in boosting algorithms is not well understood.
method Presented a unified framework for gradient and Newton boosting, and compared them with tree base learners.
result Newton boosting outperforms gradient and hybrid boosting in predictive accuracy on most datasets.
Tree-structured boosting connects gradient boosted stumps and full decision trees.
problem Connecting gradient boosted stumps and full decision trees.
method Introducing tree-structured boosting to create a single decision tree.
result Tree-structured boosting produces models equivalent to CART or gradient boosted stumps at the extremes.