arXiv research

The study investigates kernel-target alignment in tree ensemble kernels.

In this paper, we propose the distributed tree kernels (DTK) as a novel method to reduce time and space complexity of tree kernels. Using a linear complexity algorithm to compute vectors for trees, we embed feature spaces of tree fragments in low-dimensional spaces where the kernel computation is directly done with dot…

Tree ensembles like RF and GBT can be seen as kernels, improving regression and classification performance.

Kauri is a novel unsupervised binary tree for clustering that outperforms existing methods.

Tree++ graph kernel captures similarities at multiple granularities.

The kernel method is a potential approach to analyzing structured data such as sequences, trees, and graphs; however, unordered trees have not been investigated extensively. Kimura et al. (2011) proposed a kernel function for unordered trees on the basis of their subpaths, which are vertical substructures of trees resp…

BARK optimizes black-box functions using Bayesian Additive Regression Trees.

Paper analyzes soft tree ensembles using NTK, finding only leaf count matters.

We introduce a novel boosting algorithm called `KTBoost' which combines kernel boosting and tree boosting. In each boosting iteration, the algorithm adds either a regression tree or reproducing kernel Hilbert space (RKHS) regression function to the ensemble of base learners. Intuitively, the idea is that discontinuous …

TREX explains tree ensembles by identifying key training examples.

The recently proposed "generalized min-max" (GMM) kernel can be efficiently linearized, with direct applications in large-scale statistical learning and fast near neighbor search. The linearized GMM kernel was extensively compared in with linearized radial basis function (RBF) kernel. On a large number of classificatio…

While tree methods have been popular in practice, researchers and practitioners are also looking for simple algorithms which can reach similar accuracy of trees. In 2010, (Ping Li UAI'10) developed the method of "abc-robust-logitboost" and compared it with other supervised learning methods on datasets used by the deep …

Develops RKHS framework for analyzing tree ensembles.

We introduce the Mondrian kernel, a fast random feature approximation to the Laplace kernel. It is suitable for both batch and online learning, and admits a fast kernel-width-selection procedure as the random features can be re-used efficiently for all kernel widths. The features are constructed by sampling trees via a…

Develops kernels for matchings, overcoming computational challenges.

This paper introduces TNTK to study infinite soft tree ensembles.

Decision forests are widely used for classification and regression tasks. A lesser known property of tree-based methods is that one can construct a proximity matrix from the tree(s), and these proximity matrices are induced kernels. While there has been extensive research on the applications and properties of kernels, …

pGMM kernel outperforms ordinary ridge regression and RBF kernel ridge regression without tuning.

Any-gram kernels are a flexible and efficient way to employ bag-of-n-gram features when learning from textual data. They are also compatible with the use of word embeddings so that word similarities can be accounted for. While the original any-gram kernels are implemented on top of tree kernels, we propose a new approa…

Kernel density estimation (KDE) is a popular statistical technique for estimating the underlying density distribution with minimal assumptions. Although they can be shown to achieve asymptotic estimation optimality for any input distribution, cross-validating for an optimal parameter requires significant computation do…

Tree ensemble kernels improve Bayesian optimization for mixed features and constraints.

Paper proposes a novel SVM method for creating survival trees.

Tree data are ubiquitous because they model a large variety of situations, e.g., the architecture of plants, the secondary structure of RNA, or the hierarchy of XML files. Nevertheless, the analysis of these non-Euclidean data is difficult per se. In this paper, we focus on the subtree kernel that is a convolution kern…

A new kernel-based CI test improves on existing methods.

New methods improve tree ensemble models by compressing them while maintaining accuracy.

We present a generalization of independent component analysis (ICA), where instead of looking for a linear transform that makes the data components independent, we look for a transform that makes the data components well fit by a tree-structured graphical model. Treating the problem as a semiparametric statistical prob…

Optimal transport for measures on noisy tree metrics is solved with robust approach.

One-class Classification (OCC) is an area of machine learning which addresses prediction based on unbalanced datasets. Basically, OCC algorithms achieve training by means of a single class sample, with potentially some additional counter-examples. The current OCC models give satisfaction in terms of performance, but th…

P-Trees improve investment performance by optimizing the efficient frontier.

A hybrid model for Bayesian optimization handles mixed variables using MCTS for categorical and GP for continuous.

Study on embedding tree products into groups, distinguishing them.

Functional central limit theorem for kernel gradient flow and infinitesimal gradient boosting

Many machine learning models can be attacked with adversarial examples, i.e. inputs close to correctly classified examples that are classified incorrectly. However, most research on adversarial attacks to date is limited to vectorial data, in particular image data. In this contribution, we extend the field by introduci…

Improved DP KDE with better privacy and efficiency.

There is a forgetful map from the mapping class group of a punctured surface to that of the surface with one fewer puncture. We prove that finitely generated purely pseudo-Anosov subgroups of the kernel of this map are convex cocompact in the sense of B. Farb and L. Mosher. In particular, we obtain an affirmative answe…

High density clusters can be characterized by the connected components of a level set $L(λ) = \{x:\ p(x)>λ\}$ of the underlying probability density function $p$ generating the data, at some appropriate level $λ\geq 0$. The complete hierarchical clustering can be characterized by a cluster tree ${\cal T}= \bigcup_λ L(λ)…

Bayesian Additive Distribution Regression (DistBART) predicts distributions from grouped data.

TransBoost improves financial inclusion by evaluating individual financial risk.

New KD-tree based method for private synthetic data generation.

Group fairness is an important concern for machine learning researchers, developers, and regulators. However, the strictness to which models must be constrained to be considered fair is still under debate. The focus of this work is on constraining the expected outcome of subpopulations in kernel regression and, in part…

We analyze the consistency of decision trees and random forests in regression.

Paper introduces a new, tractable measure of model complexity.

Tree structured graphical models are powerful at expressing long range or hierarchical dependency among many variables, and have been widely applied in different areas of computer science and statistics. However, existing methods for parameter estimation, inference, and structure learning mainly rely on the Gaussian or…

We derive and analyze a generic, recursive algorithm for estimating all splits in a finite cluster tree as well as the corresponding clusters. We further investigate statistical properties of this generic clustering algorithm when it receives level set estimates from a kernel density estimator. In particular, we derive…

Unified framework for selecting variables with uncertainty quantification.

Enhances random forests by smoothing predictions for better performance.

Fairness, through its many forms and definitions, has become an important issue facing the machine learning community. In this work, we consider how to incorporate group fairness constraints in kernel regression methods, applicable to Gaussian processes, support vector machines, neural network regression and decision t…

Improved GAS models using trees and forests for better forecasts.