BoostTransformer uses boosting to improve transformer efficiency and accuracy.
problem Heavy computational resources and hyperparameter tuning in transformer architectures.
method Augments transformers with boosting principles through subgrid token selection and importance-weighted sampling, incorporating a least square boosting objective directly into the pipeline.
result BoostTransformer demonstrates faster convergence and higher accuracy compared to standard transformers.
ABHT boosts regression by filtering regions with different smoothness.
problem Improving regression performance through local adaptivity.
method Gradient boosting with adaptive histogram transform.
result ABHT converges faster than PEHT in Hölder continuous spaces.
Gradient Boosted Normalizing Flows improve flexibility of NFs without increasing complexity.
problem Improving flexibility of normalizing flows without increasing complexity.
method Gradient Boosting applied to normalizing flows to create a mixture model structure.
result GBNFs outperform non-boosted NFs and produce better results with simpler components.
GBHT uses gradient boosting for density estimation with theoretical guarantees.
problem Density estimation for unsupervised learning.
method Gradient Boosting Histogram Transform (GBHT) with Negative Log Likelihood loss.
result GBHT achieves faster convergence rates and better performance than base learners in density estimation.
Boosting methods for interval-censored data improve predictive accuracy in survival analysis.
problem Handling interval-censored data in survival analysis and time-to-event studies.
method Nonparametric boosting methods using censoring unbiased transformations and functional gradient descent.
result Effective boosting methods for regression and classification with interval-censored data, offering robust performance.
Gradient boosting can be seen as Gaussian process inference.
problem Improving uncertainty estimates in out-of-domain detection.
method Gradient boosting reformulated as a kernel method converging to Gaussian process inference.
result Gradient boosting can provide better uncertainty estimates through Monte-Carlo estimation of posterior variance.
Transformers fine-tuned on synthetic data boost tabular data classification performance.
problem Improving tabular data classification accuracy.
method Fine-tuning ICL-transformers on synthetic datasets with complex decision boundaries.
result Fine-tuned ICL-transformers outperform regular neural networks on real-world datasets.
A censored transformed model for proportional outcomes with boundary mass and an application to loss given default modeling.
problem Modeling proportional outcomes with boundary mass in loss given default (LGD) modeling.
method Zero-one censored transformed normal (ZOC-TN) model.
result Captures a wider range of qualitative density shapes than benchmark models while being parsimonious, computationally efficient, and numerically stable.
Randomized positional encodings boost transformer performance on longer sequences.
problem Transformers struggle with generalizing to sequences of arbitrary length.
method Introduced randomized positional encodings that simulate longer sequences and randomly select positions.
result Randomized positional encodings increase test accuracy by 12.0% on average for sequences of unseen length.
Multi-output prediction deals with the prediction of several targets of possibly diverse types. One way to address this problem is the so called problem transformation method. This method is often used in multi-label learning, but can also be used for multi-output prediction due to its generality and simplicity. In thi…
Random forest models systematically bias predictions; a numerical transform corrects this.
problem Systematic bias in random forest regression predictions.
method Defined a numerical transformation using training data to correct bias.
result Improved predictions across real-world and synthetic datasets.
A new boosting method reduces overfitting and negative transfer in transfer learning.
problem Overfitting and negative transfer in transfer learning.
method Importance sampling in boosting and random-forest based ensemble methodology.
result Performs better than competitive transfer learning methodologies 63% of the time.
Boosts linear neurons for faster convergence in neural networks.
problem Optimizing neural networks' neurons in function space.
method Reparameterizing network to whiten neuron features using Boosted Backpropagation.
result Equivalent to preconditioned gradient descent, leading to faster convergence.
This work provides theoretical and empirical evidence that invariance-inducing regularizers can increase predictive accuracy for worst-case spatial transformations (spatial robustness). Evaluated on these adversarially transformed examples, we demonstrate that adding regularization on top of standard or adversarial tra…
In this paper we propose a method to build a neural network that is similar to an ensemble of decision trees. We first illustrate how to convert a learned ensemble of decision trees to a single neural network with one hidden layer and an input transformation. We then relax some properties of this network such as thresh…
Researchers resolved ambiguities in gravitational radiation charges.
problem Ambiguities in charges related to gravitational radiation.
method Addressed supertranslation ambiguities in classical and extended BMS algebras.
result Proposed and proved an invariant charge free from supertranslation ambiguity.
The paper proposes blending gradient boosted trees and neural networks for hierarchical time series forecasting.
problem Point and probabilistic forecasting of hierarchical time series.
method A blending methodology of gradient boosted trees and neural networks, with feature engineering and diverse model selection.
result Ranked within the gold medal range in both Accuracy and Uncertainty tracks of the M5 Competition.
Boosting as gradient descent algorithms is one popular method in machine learning. In this paper a novel Boosting-type algorithm is proposed based on restricted gradient descent with structural sparsity control whose underlying dynamics are governed by differential inclusions. In particular, we present an iterative reg…
Algorithm infers sampling distribution from i.i.d. samples without supervision.
problem Learning probability distributions from unlabeled data.
method Unsupervised tree boosting using additive tree ensembles and new distributional operations.
result Algorithm outperforms deep learning in multivariate density estimation.
MultiRocket boosts TSC speed and accuracy with pooling and transformations.
problem Efficient time series classification with high accuracy.
method Multiple pooling operators and transformations applied to raw and differenced series.
result MultiRocket outperforms MiniRocket and is competitive with state-of-the-art methods in terms of accuracy and speed.
Single neural network predicts ImageNet model parameters for faster training.
problem Training diverse ImageNet models requires significant resources and time.
method Trained a neural network to predict ImageNet model parameters and used them for initialization.
result Models initialized with predicted parameters converge faster and achieve competitive performance.
LGV boosts adversarial attacks by improving surrogate models.
problem Improving the transferability of black-box adversarial attacks.
method LGV uses a pretrained surrogate model and multiple weight sets from additional training epochs to generate an effective surrogate ensemble.
result LGV outperforms other test-time transformations by significant margins.
GATES improves neural architecture search by modeling operations as information transformation.
problem Improving predictor-based neural architecture search efficiency.
method GATES models operations as information transformation, covering both node and edge cell search spaces.
result GATES boosts sample efficiency and improves predictor performance.
Simple GBRT model improved by window-based input transformation outperforms state-of-the-art deep learning models.
problem Improving performance of traditional forecasting models for time series data.
method Transformed GBRT model input structure to include target values and external features, forming one input instance per training window.
result Simple GBRT model with window-based input transformation outperformed state-of-the-art deep learning models on nine datasets.
A hybrid framework uses machine learning to price options faster and more accurately.
problem Rapid recalibration of option pricing models in dynamic markets.
method Integrates smooth offset algorithm with supervised machine learning models.
result Surrogate pricing operators achieve up to 1000x speedup over direct SOA evaluation.
GCWSNet improves neural network training speed and accuracy with power transformation.
problem Training deep neural networks efficiently and accurately.
method Developed GCWS for hashing powered-GMM kernel, enabling power transformation on data.
result GCWSNet often improves classification accuracy and converges faster with one epoch.
Fair MP-Boost improves fairness and interpretability in boosting methods.
problem Improving fairness and interpretability in boosting methods.
method Fair MP-Boost uses adaptive sampling of minipatches to balance accuracy and fairness.
result Fair MP-Boost enhances fairness and accuracy while providing interpretable feature importance.
We apply generative adversarial network (GAN) technology to build an event generator that simulates particle production in electron-proton scattering that is free of theoretical assumptions about underlying particle dynamics. The difficulty of efficiently training a GAN event simulator lies in learning the complicated …
Models predict Alzheimer's Dementia from spontaneous speech with high accuracy.
problem Early diagnosis of Alzheimer's Dementia (AD) through spontaneous speech analysis.
method Compared natural language processing techniques including SVM, GBDT, CRFs, and Transformer-based models.
result Top models achieve 0.81-0.82 test set scores for AD vs controls and 4.58 RMSE for Mental Mini State Exam scores.
MP-Boost boosts accuracy faster and more interpretable than AdaBoost.
problem Developing a faster, more interpretable boosting method.
method Adaptive selection of minipatches (small subsets of instances and features) at each iteration.
result Achieves comparable accuracy to AdaBoost and gradient boosting but faster and more interpretable.
Gradient Boosting Decision Trees (GBDTs) have become very successful in recent years, with many awards in machine learning and data mining competitions. There have been several recent studies on how to train GBDTs in the federated learning setting. In this paper, we focus on horizontal federated learning, where data sa…
Boost-R uses gradient boosted trees for analyzing recurrence data.
problem Analyzing recurrence data with static and dynamic features.
method Gradient boosted additive trees with time-dependent functions.
result Estimates the cumulative intensity function of recurrent event processes.
Boosted decision trees typically yield good accuracy, precision, and ROC area. However, because the outputs from boosting are not well calibrated posterior probabilities, boosting yields poor squared error and cross-entropy. We empirically demonstrate why AdaBoost predicts distorted probabilities and examine three cali…
Hydra boosts efficiency for long-context reasoning in resource-constrained settings.
problem Quadratic complexity of transformers limits long-context reasoning in resource-constrained systems.
method Hydra uses a modular architecture with adaptive routing between sparse global attention, mixture-of-experts, and dual memories.
result Hydra achieves significant throughput and accuracy improvements for long-context reasoning.
Extends boosting to multiclass online agnostic classification.
problem Online multiclass classification with weak learners.
method Reduces multiclass online agnostic boosting to online convex optimization.
result First boosting algorithm for online agnostic multiclass classification.
Boosting is one of the most significant developments in machine learning. This paper studies the rate of convergence of L2Boosting, which is tailored for regression, in a high-dimensional setting. Moreover, we introduce so-called \textquotedblleft post-Boosting\textquotedblright. This is a post-selection estimator w…
Paper proposes SERT model for US stock pricing, outperforming standard models during market shocks.
problem Capturing patterns of temporal sparsity in asset pricing during market fluctuations.
method Introduces SERT model based on pre-trained Transformer, compares with standard models in three periods.
result SERT model achieves highest out-of-sample R2 (11.94\% and 11.47\%) during extreme market fluctuations. We present a new boosting algorithm, motivated by the large margins theory for boosting. We give experimental evidence that the new algorithm is significantly more robust against label noise than existing boosting algorithm.
Digitwashing gap boosts stock crash risk, study finds.
problem The gap between companies' digital promises and actual performance increases stock crash risk.
method Empirical analysis of Shanghai and Shenzhen A-share companies from 2010 to 2021, robustness tests conducted.
result GDT significantly increases stock price crash risk, confirmed by robust tests.
Improved agnostic boosting with better sample efficiency.
problem Agnostic boosting's poor sample efficiency compared to Empirical Risk Minimization.
method Leverages sample reuse across rounds, guarantees better generalization.
result Substantially more sample-efficient agnostic boosting algorithm.
Excellent ranking power along with well calibrated probability estimates are needed in many classification tasks. In this paper, we introduce a technique, Calibrated Boosting-Forest that captures both. This novel technique is an ensemble of gradient boosting machines that can support both continuous and binary labels. …
Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP), achieving state-of-the-art results in domains such as language modeling and machine t…
Boosting algorithms are frequently used in applied data science and in research. To date, the distinction between boosting with either gradient descent or second-order Newton updates is often not made in both applied and methodological research, and it is thus implicitly assumed that the difference is irrelevant. The g…
The study provides statistical guarantees for Bayesian variational boosting.
problem Statistical and convergence issues in variational boosting.
method Proposed a novel variational family and a functional Frank-Wolfe optimization algorithm.
result Demonstrated stochastic boundedness and provided convergence rate for boosting iterates.
In this paper we propose using the principle of boosting to reduce the bias of a random forest prediction in the regression setting. From the original random forest fit we extract the residuals and then fit another random forest to these residuals. We call the sum of these two random forests a \textit{one-step boosted …
TF Boosted Trees (TFBT) is a new open-sourced frame-work for the distributed training of gradient boosted trees. It is based on TensorFlow, and its distinguishing features include a novel architecture, automatic loss differentiation, layer-by-layer boosting that results in smaller ensembles and faster prediction, princ…
Gradient boosting algorithms construct a regression predictor using a linear combination of ``base learners''. Boosting also offers an approach to obtaining robust non-parametric regression estimators that are scalable to applications with many explanatory variables. The robust boosting algorithm is based on a two-stag…
As an adaptive, interpretable, robust, and accurate meta-algorithm for arbitrary differentiable loss functions, gradient tree boosting is one of the most popular machine learning techniques, though the computational expensiveness severely limits its usage. Stochastic gradient boosting could be adopted to accelerates gr…