ID3 generates near-optimal decision trees for DNFs under product distributions.
problem Understanding the optimality of decision trees generated by ID3.
method Introducing a new metric (MIC) to measure the optimality of ID3-generated trees and comparing it with other algorithms.
result The TopDown variant of ID3 is near-optimal in learning read-once DNFs under product distributions, while another variant is not.
New algorithms minimize MMD to approximate probability measures efficiently.
problem Approximating probability measures by representative point sets.
method Sequential greedy minimization of maximum mean discrepancy (MMD) over candidate sets, with mini-batch variants.
result Consistency of proposed algorithms and mini-batch variants established.
Paper characterizes gradient descent in high-dimensional learning problems.
problem Understanding gradient descent dynamics in high-dimensional statistical learning.
method Non-asymptotic joint distributional characterization of gradient descent iterates and debiased statistics.
result Gradient descent iterates approximate normality after debiasing correction.
The stochastic gradient descent (SGD) algorithm has been widely used in statistical estimation for large-scale data due to its computational and memory efficiency. While most existing works focus on the convergence of the objective function or the error of the obtained solution, we investigate the problem of statistica…
Develops a dynamic mean field theory for reinforcement learning.
problem Finite state and action Bayesian reinforcement learning in large state spaces.
method Analogies with statistical physics, interpreting probabilities as couplings and values as spins, solving mean field equations.
result State-action values are statistically independent in the asymptotic state space limit, with exact or approximate equations for computation.
Within a statistical learning setting, we propose and study an iterative regularization algorithm for least squares defined by an incremental gradient method. In particular, we show that, if all other parameters are fixed a priori, the number of passes over the data (epochs) acts as a regularization parameter, and prov…
We propose a novel framework for graph mean computation.
problem Defining a mean for graph data is difficult.
method Embeddings in the space of smooth graph signal distributions, using the Wasserstein metric.
result Existence and uniqueness of the graph mean established, and an iterative algorithm provided.
The mean field variational Bayes method is becoming increasingly popular in statistics and machine learning. Its iterative Coordinate Ascent Variational Inference algorithm has been widely applied to large scale Bayesian inference. See Blei et al. (2017) for a recent comprehensive review. Despite the popularity of the …
New method solves constrained stochastic optimization problems efficiently.
problem Online statistical inference of constrained stochastic nonlinear optimization problems.
method Stochastic Sequential Quadratic Programming (StoSQP) with iterative sketching solver.
result The rescaled primal-dual sequence converges to a mean-zero Gaussian distribution.
We propose a novel method to accelerate Lloyd's algorithm for K-Means clustering. Unlike previous acceleration approaches that reduce computational cost per iterations or improve initialization, our approach is focused on reducing the number of iterations required for convergence. This is achieved by treating the assig…
A new k-means method improves clustering accuracy and robustness.
problem Insufficient asymptotic properties in existing k-means methods. method Introducing clustering consistency and proposing a new k-means method. result The new k-means method has lower clustering error rates and is more robust. Enhanced TSFMs improve time series forecasting accuracy and reliability.
problem Variance, bias, and uncertainty in TSFMs' predictions on real data.
method Statistical and ensemble techniques including bagging, stacking, residual modeling, and prediction intervals.
result Hybrid models consistently outperform standalone TSFMs across multiple horizons.
Improved estimation of higher order integrals using shrinkage techniques.
problem Estimating higher order Bochner integrals in non-parametric settings.
method Shrinkage of U-statistic towards a target element, considering kernel degeneracy.
result Consistent shrinkage estimators with fast rates of convergence, even for non-degenerate kernels.
New algorithms robustly estimate mean with near-optimal error rates.
problem Outlier robust mean estimation in high-dimensional data.
method Stability condition and iterative filtering algorithms.
result Optimal error rates with subgaussian rates for robust mean estimation.
Paper provides statistical guarantees for GNNs in link prediction.
problem Link prediction accuracy in graph neural networks.
method Proposes a linear GNN architecture (LG-GNN) and derives statistical guarantees.
result LG-GNN produces consistent estimators for edge probabilities and has better detection of high-probability edges.
We lay theoretical foundations for new database release mechanisms that allow third-parties to construct consistent estimators of population statistics, while ensuring that the privacy of each individual contributing to the database is protected. The proposed framework rests on two main ideas. First, releasing (an esti…
LocalKMeans parallelizes Lloyd's algorithm for distributed data.
problem Efficiently clustering data across multiple machines.
method Parallel local iterations with synchronization every L steps.
result Higher required signal-to-noise ratio due to local steps.
Although consistency is a minimum requirement of any estimator, little is known about consistency of the mean partition approach in consensus clustering. This contribution studies the asymptotic behavior of mean partitions. We show that under normal assumptions, the mean partition approach is consistent and asymptotic …
Neural networks simplify SDR in regression tasks.
problem Sufficient dimension reduction in regression problems.
method Applying neural networks with rank regularization to estimate the central mean subspace.
result Neural networks effectively perform SDR, consistent with theoretical estimations.
Clustering is a separation of data into groups of similar objects. Every group called cluster consists of objects that are similar to one another and dissimilar to objects of other groups. In this paper, the K-Means algorithm is implemented by three distance functions and to identify the optimal distance function for c…
Identifying a set of homogeneous clusters in a heterogeneous dataset is one of the most important classes of problems in statistical modeling. In the realm of unsupervised partitional clustering, k-means is a very important algorithm for this. In this technical report, we develop a new k-means variant called Augmented …
Paper proposes an online estimator for covariance matrix of SGD iterates.
problem Quantifying variability and randomness of SGD-based estimates in online learning.
method Proposes a fully online estimator for covariance matrix of ASGD using SGD iterates.
result Establishes consistency of the online estimator and shows comparable convergence rate to offline methods.
The paper improves matrix completion with auxiliary covariates using LS estimation.
problem Matrix completion with noisy data and auxiliary covariates.
method Iterative least squares estimation with statistical properties derived.
result Asymptotic normal distributions of estimators for low-rank matrix and coefficient matrix.
New tests for binary classification regression functions without distribution assumptions.
problem Testing regression functions in binary classification without distributional assumptions.
method Conditional kernel mean embeddings and resampling-based framework.
result Distribution-free hypothesis tests with exact type I error control.
We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as…
GEORCE-FM algorithm optimizes Fréchet means and distances efficiently.
problem Computing Fréchet means on Riemannian manifolds efficiently.
method GEORCE-FM algorithm that simultaneously computes Fréchet means and distances in local charts.
result GEORCE-FM algorithm converges globally and locally quadratically, and scales to large datasets.
Method infers dynamics from incomplete time series data.
problem Challenges in inferring stochastic dynamics from time series with missing data.
method Expectation Maximization (EM) algorithm that iterates between E-step and M-step.
result The EM algorithm effectively recovers missing data points and infers underlying network models from real neuronal activities.
Paper proposes a new descriptor for early trajectory characterization in matrix iterations.
problem Comparing early behavior of high-dimensional trajectories in nonlinear matrix iterations.
method Develops a two-channel fuzzy coordinate system using F-transform for compact representation.
result The descriptor achieves high R^2 values (mean = 0.6480) in approximating convergence lengths.
Develops a computationally tractable differentially private mean estimator called the balloon mean.
problem Robust mean estimation in the presence of outliers and heavy-tailed distributions.
method Iterative clipping procedure over Mahalanobis balls.
result Balloon mean is robust to outliers and outperforms existing estimators in contaminated settings.
This paper develops algorithms for high-dimensional stochastic control problems based on deep learning and dynamic programming. Unlike classical approximate dynamic programming approaches, we first approximate the optimal policy by means of neural networks in the spirit of deep reinforcement learning, and then the valu…
A new method optimizes kernels for GANs and SVMs using mean-field theory.
problem Optimizing kernels for GANs and SVMs in a distributionally robust setting.
method Distributionally robust optimization, Monte-Carlo SAA, particle SGD, mean-field analysis.
result The method improves kernel learning for hypothesis testing and achieves better test power.
The sample mean is among the most well studied estimators in statistics, having many desirable properties such as unbiasedness and consistency. However, when analyzing data collected using a multi-armed bandit (MAB) experiment, the sample mean is biased and much remains to be understood about its properties. For exampl…
DKMD is a fast signed statistic for comparing univariate distributions.
problem Comparing univariate distributions, especially preserving directionality.
method DKMD integrates kernel mean embeddings against an odd weighting function.
result DKMD preserves directionality and is robust to outliers.
Optimizes cover parameter in Mapper algorithm for better visualization.
problem Tuning the cover parameter in Mapper algorithm to generate a ``nice'' graph.
method Optimizes cover by repeatedly splitting using statistical tests and Gaussian mixture model.
result Algorithm generates covers that retain dataset essence while being faster.
Huber regression assessed for robustness in statistical learning.
problem Understanding Huber regression in nonparametric statistical learning.
method Assessment from statistical learning perspective, focusing on risk consistency, adaptive tuning, and convergence rates.
result Huber regression can be asymptotically mean regression calibrated under (1+ε)-moment conditions, justifying its robustness. Study examines parallel computing strategies for faster imputation of missing data.
problem Time-consuming iterative imputation methods for large datasets.
method Variable-wise and model-wise distributed parallel computing strategies in missForest.
result Variable-wise distributed strategy introduces additional biases in imputation results.
We consider the problem of clustering noisy finite-length observations of stationary ergodic random processes according to their nonparametric generative models without prior knowledge of the model statistics and the number of generative models. Two algorithms, both using the L1-distance between estimated power spectra…
Paper develops robust regression method for heavy-tailed errors.
problem High-dimensional robust regression with heavy-tailed errors.
method Iteratively reweighted ℓ1-penalized adaptive Huber regression. result Oracle convergence rate and variable selection consistency achieved.
New statistical properties for mini-batch Cox-NN optimization.
problem Optimizing deep Cox neural networks using mini-batches.
method Developed mini-batch maximum partial-likelihood estimator (mb-MPLE) for Cox-NN.
result mb-MPLE is consistent and achieves optimal convergence rate.
Paper develops a unified framework for measuring differences between conditional distributions.
problem Comparing conditional distributions in a unified and theoretically sound manner.
method Kernel embeddings and conditional maximum mean discrepancy (CMMD) framework.
result Established a coherent framework for measuring divergence between conditional distributions.
SNAP improves robust computation by emphasizing trustworthy items and downweighting outliers.
problem Improving robustness in computation, especially in high-dimensional settings.
method SNAP assigns weights based on mutual agreement, suppressing outlier contributions.
result SNAP ensures outliers contribute negligibly to computations, even in high-dimensional settings.
Variational Bayes (VB), also known as independent mean-field approximation, has become a popular method for Bayesian network inference in recent years. Its application is vast, e.g. in neural network, compressed sensing, clustering, etc. to name just a few. In this paper, the independence constraint in VB will be relax…
We study the tradeoff between the statistical error and communication cost of distributed statistical estimation problems in high dimensions. In the distributed sparse Gaussian mean estimation problem, each of the m machines receives n data points from a d-dimensional Gaussian distribution with unknown mean θ w…
Efficient and robust algorithms for decentralized estimation in networks are essential to many distributed systems. Whereas distributed estimation of sample mean statistics has been the subject of a good deal of attention, computation of U-statistics, relying on more expensive averaging over pairs of observations, is…
In this note, we introduce a new algorithm to deal with finite dimensional clustering with errors in variables. The design of this algorithm is based on recent theoretical advances (see Loustau (2013a,b)) in statistical learning with errors in variables. As the previous mentioned papers, the algorithm mixes different t…
Single-step samplers generate high-quality samples efficiently.
problem Sampling from unnormalized distributions is computationally expensive.
method Developed consistent diffusion samplers that generate samples in a single step.
result Single-step samplers produce high-fidelity samples with less than 1% of traditional samplers' evaluations.
New theory explains consistency of kernel methods with non-i.i.d. data.
problem Consistency of kernel methods under non-i.i.d. data.
method Empirical weak convergence (EWC) as a general assumption.
result Established consistency of SVMs, kernel mean embeddings, and CKMEs with EWC data.
This paper studies statistical estimation in optional regression models.
problem Estimating parameters in regression models with optional semimartingale processes.
method Structural least squares (LS) estimates and their sequential versions.
result Strong consistency of LS-estimates and fixed accuracy of sequential LS-estimates.