Prototype rules simplify multiclass classification in metric spaces, achieving consistency and reduced complexity.
problem Multiclass classification in metric spaces, focusing on universal consistency and convergence rates.
method Novel Proto-NN and hybrid rules for multiclass classification in metric spaces, analyzing convergence rates.
result Proto-NN is universally consistent and simpler to implement, with similar computational advantages.
The paper optimizes k-NN for distributed learning with minimax optimal performance.
problem Minimizing error rates in classification, regression, and density estimation.
method Optimal aggregation of fixed-k nearest neighbors from multiple subsets of data.
result Achieves minimax optimal error rates up to a logarithmic factor.
Consistency of k-NN rule proven in sigma-finite dimensional metric spaces.
problem Proving consistency of k-NN rule in metric spaces.
method Direct proof using Stone's theorem, investigating metric properties.
result Universal consistency of k-NN rule in sigma-finite dimensional metric spaces.
A novel k-NN method estimates conditional mean and variance efficiently.
problem Joint estimation of conditional mean and variance.
method Integrates k-NN with automated variance selection.
result Achieves fast convergence rates and improved precision.
K-NN classifier is one of the most famous classification algorithms, whose performance is crucially dependent on the distance metric. When we consider the distance metric as a parameter of K-NN, learning an appropriate distance metric for K-NN can be seen as minimizing the empirical risk of K-NN. In this paper,…
Learning rule consistency tied to non-existence of real-valued measurable cardinals.
problem Consistency of k-NN learning rule in metric spaces.
method Analyzing separable subspaces and density conditions.
result The k-NN classifier's consistency depends on the absence of real-valued measurable cardinals.
This study combines two different learning paradigms, k-nearest neighbor (k-NN) rule, as memory-based learning paradigm and relevance vector machines (RVM), as statistical learning paradigm. This combination is performed in kernel space and is called k-relevance vector (k-RV). The purpose is to improve the performance …
Relevance vector machine (RVM) can be seen as a probabilistic version of support vector machines which is able to produce sparse solutions by linearly weighting a small number of basis functions instead using all of them. Regardless of a few merits of RVM such as giving probabilistic predictions and relax of parameter …
Paper proposes MS-k-NN for improved convergence rate in k-NN classification.
problem Improving convergence rate of k-NN classification methods.
method Proposes MS-k-NN that extrapolates unweighted k-NN estimators to k=0.
result MS-k-NN achieves improved convergence rate under certain conditions.
In the k-nearest neighborhood model (k-NN), we are given a set of points P, and we shall answer queries q by returning the k nearest neighbors of q in P according to some metric. This concept is crucial in many areas of data analysis and data processing, e.g., computer vision, document retrieval and machi…
We derive high-probability finite-sample uniform rates of consistency for k-NN regression that are optimal up to logarithmic factors under mild assumptions. We moreover show that k-NN regression adapts to an unknown lower intrinsic dimension automatically. We then apply the k-NN regression rates to establish new …
The k-nearest neighbor classification method (k-NNC) is one of the simplest nonparametric classification methods. The mutual k-NN classification method (MkNNC) is a variant of k-NNC based on mutual neighborship. We propose another variant of k-NNC, the symmetric k-NN classification method (SkNNC) based …
Implementing k-NN classification using Gromov--Wasserstein distances
problem Comparing metric measure spaces
method Gromov--Wasserstein and fused Gromov--Wasserstein distances
result Universal consistency of k-NN classifiers A fast method for LOOCV in k-NN regression reduces computation time.
problem Efficient computation of LOOCV for k-NN regression.
method Identical LOOCV estimate to (k+1)-NN MSE on training data.
result LOOCV computation can be done with (k+1)-NN regression once.
The k-nearest neighbour (k-NN) classifier is one of the oldest and most important supervised learning algorithms for classifying datasets. Traditionally the Euclidean norm is used as the distance for the k-NN classifier. In this thesis we investigate the use of alternative distances for the k-NN classifier. We …
Study shows k-NN classifier is not universally consistent on (0,1) but consistent on discrete and specific measure spaces.
problem Consistency of k-NN classifier under Wasserstein distance on measure spaces. method Analysis of k-NN classifier properties under Wasserstein distance, use of σ-finite metric dimension, geodesic structures of Wasserstein spaces. result Consistency of k-NN classifier on specific measure spaces (discrete, Gaussian, wavelet series) but not on (0,1). Improved OOD detection using label smoothing and k-NN density estimates.
problem Detecting out-of-distribution examples in classification models.
method Label smoothing and k-NN density estimate on intermediate activations.
result Label smoothing improves OOD detection performance, both theoretically and empirically.
Multiple classifier systems focus on the combination of classifiers to obtain better performance than a single robust one. These systems unfold three major phases: pool generation, selection and integration. One of the most promising MCS approaches is Dynamic Selection (DS), which relies on finding the most competent c…
The k-NN graph has played a central role in increasingly popular data-driven techniques for various learning and vision tasks; yet, finding an efficient and effective way to construct k-NN graphs remains a challenge, especially for large-scale high-dimensional data. In this paper, we propose a new approach to const…
We investigate the classification performance of K-nearest neighbors (K-NN) and deep neural networks (DNNs) in the presence of label noise. We first show empirically that a DNN's prediction for a given test example depends on the labels of the training examples in its local neighborhood. This motivates us to derive a r…
In the k-nearest neighbor algorithm (k-NN), the determination of classes for test instances is usually performed via a majority vote system, which may ignore the similarities among data. In this research, the researcher proposes an approach to fine-tune the selection of neighbors to be passed to the majority vote syste…
We propose a simple approach which, given distributed computing resources, can nearly achieve the accuracy of k-NN prediction, while matching (or improving) the faster prediction time of 1-NN. The approach consists of aggregating denoised 1-NN predictors over a small number of distributed subsamples. We show, bot…
Paper develops robust k-NN algorithm for few samples.
problem Learning robust classifier from limited samples.
method Distributionally robust formulation of weighted k-NN. result Robust classifier improves generalization with smaller Lipschitz norm.
Previously, we proposed a physically inspired rule to organize the data points in a sparse yet effective structure, called the in-tree (IT) graph, which is able to capture a wide class of underlying cluster structures in the datasets, especially for the density-based datasets. Although there are some redundant edges or…
Simple k-NN filtering improves model accuracy on noisy labels.
problem Training models with noisy labels reduces performance and is hard to identify.
method A simple k-nearest neighbor-based filtering approach on the logit layer. result Improves model accuracy compared to recent methods.
This paper studies the relationship between the classification performed by deep neural networks (DNNs) and the decision of various classical classifiers, namely k-nearest neighbours (k-NN), support vector machines (SVM) and logistic regression (LR), at various layers of the network. This comparison provides us with ne…
k Nearest Neighbors (kNN) is one of the most widely used supervised learning algorithms to classify Gaussian distributed data, but it does not achieve good results when it is applied to nonlinear manifold distributed data, especially when a very limited amount of labeled samples are available. In this paper, we pro…
Improved convergence rate for kNN graph Laplacians with adaptive bandwidth.
problem Enhancing the efficiency of graph-based data analysis methods.
method Introducing a new class of kNN graph with adaptive bandwidth and proving operator convergence rate.
result Operator convergence rate of O(N−2/(d+6)) for the kNN graph Laplacian, up to a log factor. Improves k-NN for monotonic data with robustness against noise.
problem Class noise in real-life data violates monotonic constraints in k-NN.
method Monotonic Fuzzy k-NN (MonFkNN) with new fuzzy membership calculation.
result Significant accuracy improvements and robustness against monotonic noise.
The problem of supervised classification (or discrimination) with functional data is considered, with a special interest on the popular k-nearest neighbors (k-NN) classifier. First, relying on a recent result by Cerou and Guyader (2006), we prove the consistency of the k-NN classifier for functional data whose distribu…
Proposes LRR and LRLR for improving stock prediction accuracy.
problem Improving stock prediction accuracy through nonparametric classification.
method Local radial regression and logistic regression variant.
result LRLR outperforms LPoR and MS-k-NN in real-world stock datasets. Gatherings of thousands to millions of people frequently occur for an enormous variety of events, and automated counting of these high-density crowds is useful for safety, management, and measuring significance of an event. In this work, we show that the regularly accepted labeling scheme of crowd density maps for trai…
Estimating entropy and mutual information consistently is important for many machine learning applications. The Kozachenko-Leonenko (KL) estimator (Kozachenko & Leonenko, 1987) is a widely used nonparametric estimator for the entropy of multivariate continuous random variables, as well as the basis of the mutual inform…
PAC-Bayesian bounds improve understanding of K-NN classifier performance.
problem Improving the understanding of K-NN classifier's generalization error.
method PAC-Bayesian analysis applied to K-NN classifier in kernel space.
result PAC-Bayesian bounds provide a function of the number of redundant training examples.
Introduction. Case Based Reasoning (CBR) is an emerg- ing decision making paradigm in medical research where new cases are solved relying on previously solved similar cases. Usually, a database of solved cases is provided, and every case is described through a set of attributes (inputs) and a label (output). Extracting…
Paper improves k-NN predictive performance with efficient variable selection.
problem Improving predictive performance of k-NN models. method Efficient forward selection of predictor variables.
result Novel approach approaches outperformance of stepwise selection models.
Paper uses K-NN resampling to simulate and evaluate LOB markets.
problem Simulating and evaluating limit order book (LOB) markets.
method Applies K-nearest neighbor (K-NN) resampling to LOB simulation and evaluation. result Demonstrates the effectiveness and efficiency of K-NN resampling in LOB simulation and evaluation. Study shows SNN graph Laplacians converge to k-NN graph Laplacians under large scale asymptotics.
problem Understanding the convergence of SNN graph Laplacians to k-NN graph Laplacians.
method Analyzing the asymptotic behavior of SNN and k-NN graph Laplacians.
result The graph Laplacians of SNN and k-NN graphs converge to the same limit under large scale asymptotics.
In this paper, we consider the problem of malware detection and classification based on image analysis. We convert executable files to images and apply image recognition using deep learning (DL) models. To train these models, we employ transfer learning based on existing DL models that have been pre-trained on massive …
We propose a non-parametric anomaly detection algorithm for high dimensional data. We score each datapoint by its average K-NN distance, and rank them accordingly. We then train limited complexity models to imitate these scores based on the max-margin learning-to-rank framework. A test-point is declared as an anomaly…
We introduce a variant of the k-nearest neighbor classifier in which k is chosen adaptively for each query, rather than supplied as a parameter. The choice of k depends on properties of each neighborhood, and therefore may significantly vary between different points. (For example, the algorithm will use larger $k…
We investigate nearest neighbor and generative models for transferring pose between persons. We take in a video of one person performing a sequence of actions and attempt to generate a video of another person performing the same actions. Our generative model (pix2pix) outperforms k-NN at both generating corresponding f…
A new strategy selects k in k-NN regression without hold-out data.
problem Choosing optimal k in k-NN regression without hold-out data.
method Iterative procedure over k, minimum discrepancy principle.
result Minimax-optimal over smoothness function classes.
From a fresh data science perspective, this thesis discusses the prediction of coronary artery disease based on genetic variations at the DNA base pair level, called Single-Nucleotide Polymorphisms (SNPs), collected from the Ontario Heart Genomics Study (OHGS). First, the thesis explains two commonly used supervised le…
This paper introduces a class of k-nearest neighbor (k-NN) estimators called bipartite plug-in (BPI) estimators for estimating integrals of non-linear functions of a probability density, such as Shannon entropy and Rényi entropy. The density is assumed to be smooth, have bounded support, and be uniformly bounded from…
A new k-NN algorithm using surprisal for robust and interpretable nonparametric learning.
problem Complex patterns and relationships in data without strong distribution assumptions.
method Surprisal-driven k-NN framework for classification, regression, density estimation, and anomaly detection. result State-of-the-art results in classification and anomaly detection, competitive regression results.
Improved k-NN active learning with local smoothness assumption.
problem Active learning convergence rates under smoothness assumptions.
method Designing an active learning algorithm with better convergence rate using local smoothness assumption for k-NN.
result Better convergence rate than in passive learning.
We develop a prediction-based prescriptive model for learning optimal personalized treatments for patients based on their Electronic Health Records (EHRs). Our approach consists of: (i) predicting future outcomes under each possible therapy using a robustified nonlinear model, and (ii) adopting a randomized prescriptiv…