Many datasets are in the form of tables of binned data. Performing regression on these data usually involves either reading off bin heights, ignoring data from neighbouring bins or interpolating between bins thus over or underestimating the true bin integrals. In this paper we propose an elegant method for performing G…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “data binning”
Histogram binning method proven with guarantees without splitting data.
problem Proving theoretical guarantees for histogram binning without sample splitting.
method Using Markov property of order statistics to prove calibration guarantees for original method.
result Proves histogram binning has strong calibration guarantees without sample splitting.
Isotonic regression binning affects calibration statistics of machine learning models.
problem Isotonic regression binning introduces aleatoric uncertainty in calibration statistics.
method Calibration error statistics are recalibrated using isotonic regression, which produces stratified uncertainties.
result Stratified uncertainties lead to significant differences in bin-based calibration statistics.
New methods reduce bias in estimating calibration error.
problem Reducing bias in estimating calibration error.
method Synthesizing model outputs and using equal-mass bins.
result Two reliable calibration-error estimators found: debiased estimator and ECE_sweep.
Bin Packing problems have been widely studied because of their broad applications in different domains. Known as a set of NP-hard problems, they have different vari- ations and many heuristics have been proposed for obtaining approximate solutions. Specifically, for the 1D variable sized bin packing problem, the two ke…
New methods improve estimation of nonhomogeneous Poisson processes from limited data.
problem Estimating nonhomogeneous Poisson processes from limited data.
method Formulated as a learning generalization problem, proposed adaptive and data-driven binning methods.
result Improved estimation of nonhomogeneous Poisson processes with limited data.
A new method detects and displays pairwise dependence between variates.
problem Detecting and visualizing dependence between variates of different types.
method Recursive random binning with approximations to Pearson's statistic.
result The method is well-calibrated and powerful against common test alternatives.
Paper analyzes ECE bias and provides bounds for its estimation.
problem Understanding the estimation bias in ECE for machine learning models.
method Information-theoretic approach to analyze bias in uniform mass and uniform width binning strategies.
result Established upper bounds on ECE estimation bias and optimal number of bins.
Improved binning technique boosts nUV measure performance.
problem Improving the performance of the nUV measure in real applications.
method Introduced the nUV measure, provided theoretical optimal binning techniques, and proposed algorithms for approximate solutions.
result Approximate binning techniques show 4-13% increase in AUC scores with statistical significance.
A method for non-parametric conditional distribution estimation using CRPS-optimal binning.
problem Non-parametric conditional distribution estimation.
method Partitioning covariate-sorted observations into bins to minimize LOO-CRPS, selecting K by K-fold cross-validation of test CRPS.
result Produces narrower prediction intervals with near-nominal coverage compared to split-conformal competitors.
The MAP-Elites algorithm produces a set of high-performing solutions that vary according to features defined by the user. This technique has the potential to be a powerful tool for design space exploration, but is limited by the need for numerous evaluations. The Surrogate-Assisted Illumination algorithm (SAIL), introd…
The Effectiveness of Discretization in Forecasting: An Empirical Study on Neural Time Series Modelscs.LG
This study examines how discretization improves neural forecasting models.
problem Improving predictive performance of neural forecasting models.
method Empirical investigation of data binning techniques on various neural forecasting architectures.
result Data binning almost always improves forecasting accuracy, but the type of binning is less important.
Balls-and-Bins sampling improves DP-SGD privacy and utility.
problem Improving privacy and utility in DP-SGD implementations.
method Introducing Balls-and-Bins sampling as an alternative to shuffling in DP-SGD.
result Balls-and-Bins sampling achieves utility comparable to shuffling while offering better privacy amplification.
Distribution-free binary classification: prediction sets, confidence intervals and calibrationstat.ML
Study three types of uncertainty quantification for binary classification without distributional assumptions.
problem Uncertainty quantification for binary classification in a distribution-free setting.
method Established theorems connecting calibration, confidence intervals, and prediction sets for score-based classifiers.
result Distribution-free calibration is only possible using scoring functions that partition feature space into countably many sets.
Improved kernel ridge regression for large datasets using weighted random binning.
problem Efficiently approximating kernel matrices for large-scale datasets.
method Introduced weighted random binning features for locality sensitive hashing.
result Weighted random binning features generate Gaussian processes of any desired smoothness.
The objective of this work is to take advantage of deep neural networks in order to make next day crime count predictions in a fine-grain city partition. We make predictions using Chicago and Portland crime data, which is augmented with additional datasets covering weather, census data, and public transportation. The c…
Object detection in streaming images is a major step in different detection-based applications, such as object tracking, action recognition, robot navigation, and visual surveillance applications. In mostcases, image quality is noisy and biased, and as a result, the data distributions are disturbed and imbalanced. Most…
Moment Unfoldinghep-ph
New method unfolds distribution moments directly from data without binning.
problem Deconvolving detector distortions in particle physics.
method Uses machine learning, inspired by GANs, to unfold moments directly.
result More precise than bin-based approaches and comparable to unbinned methods.
PDE-Foam - a probability-density estimation method using self-adapting phase-space binningphysics.data-an
Probability Density Estimation (PDE) is a multivariate discrimination technique based on sampling signal and background densities defined by event samples from data or Monte-Carlo (MC) simulations in a multi-dimensional phase space. In this paper, we present a modification of the PDE method that uses a self-adapting bi…
The optimal binning is the optimal discretization of a variable into bins given a discrete or continuous numeric target. We present a rigorous and extensible mathematical programming formulation for solving the optimal binning problem for a binary, continuous and multi-class target type, incorporating constraints not p…
A new DP algorithm improves privacy in hashing and sampling for search and learning.
problem Improving privacy in hashing and sampling for large-scale applications.
method Combines differential privacy with one permutation hashing and bin-wise consistent weighted sampling.
result Proposes DP-OPH and DP-BCWS algorithms that enhance privacy while maintaining utility.
Recent developments have linked causal inference with Algorithmic Information Theory, and methods have been developed that utilize Conditional Kolmogorov Complexity to determine causation between two random variables. We present a method for inferring causal direction between continuous variables by using an MDL Binnin…
This paper improves multi-class calibration methods using mutual information maximization-based binning.
problem Calibration of deep neural network predictions, especially for small prior classes.
method I-Max concept for binning, shared class-wise calibration strategy.
result Improves multi-class ranking and calibration performance using a small calibration set.
A new survival analysis method eliminates hyperparameter tuning.
problem Survival analysis model hyperparameters are difficult to tune.
method Flexible survival density estimation with importance sampling.
result Matches or outperforms existing methods on real-world datasets.
This paper introduces minimum-risk recalibration for probabilistic classifiers, improving their reliability and accuracy.
problem Improving the reliability and accuracy of probabilistic classifiers.
method Minimum-risk recalibration within the MSE decomposition framework, analyzing UMB method and label shift adaptation.
result The optimal number of bins for UMB scales with , resulting in a risk bound of approximately .
AdaCat improves density estimation and planning in autoregressive models.
problem Efficiently modeling sharp density changes in continuous data.
method Adaptive Categorical Discretization (AdaCat) for autoregressive models.
result Improves density estimation and planning in various data types.
A new method improves quantile regression for high-dimensional data.
problem Handling heteroscedastic, multimodal, or skewed data in quantile regression.
method Dynamic prototypes-based probability density estimation with conformalized high-density quantile regression.
result Enhanced prediction regions with valid coverage guarantees and scalability to higher dimensions.
Unified framework for PDF estimation using MDL-based binning and tensor factorization.
problem Challenges in estimating PDFs for non-uniform, multimodal data.
method MDL-based binning with quantile cuts, tensor factorization (CPD).
result Effective PDF estimation on synthetic and real data.
BiN normalizes financial time-series for better forecasting.
problem Non-stationarity and multimodality in financial time-series data.
method Bilinear Normalization (BiN) incorporated into TABL networks.
result BiN-TABL outperforms other normalization methods in financial forecasting.
TCE measures calibration error with a test-based approach.
problem Measuring calibration error of probabilistic binary classifiers.
method TCE uses a novel loss function based on a statistical test.
result TCE offers clear interpretation, consistent scale, and enhanced visual representation.
In the framework of Multifractal Diffusion Entropy Analysis we propose a method for choosing an optimal bin-width in histograms generated from underlying probability distributions of interest. The method presented uses techniques of Rényi's entropy and the mean squared error analysis to discuss the conditions under whi…
New bin-wise scaling methods improve prediction uncertainty calibration for machine learning.
problem Improving prediction uncertainty calibration for machine learning regression.
method Adaptations of Binwise Variance Scaling (BVS) with alternative loss functions and feature-based binning.
result Improved adaptivity and consistency in prediction uncertainty calibration.
In subgroup discovery, also known as supervised pattern mining, discovering high quality one-dimensional subgroups and refinements of these is a crucial task. For nominal attributes, this is relatively straightforward, as we can consider individual attribute values as binary features. For numerical attributes, the task…
Smart bin monitors predict medication adherence with high accuracy.
problem Predicting chronic medication adherence to improve healthcare efficiency.
method Machine learning models trained on IoT device data.
result High predictive performance (ROC AUC 0.86).
Solves online 3D bin packing with deep reinforcement learning under constraints.
problem Challenges of packing items immediately without information and constraints.
method Constrained deep reinforcement learning (DRL) with feasibility predictor.
result Significantly outperforms state-of-the-art methods in online 3D bin packing.
Collider data must be corrected for detector effects ("unfolded") to be compared with many theoretical calculations and measurements from other experiments. Unfolding is traditionally done for individual, binned observables without including all information relevant for characterizing the detector response. We introduc…
HFNO enhances interpretability of turbulent flows through parallel wavenumber bin processing.
problem Opaque inner workings of Fourier Neural Operators (FNOs) hinder physical interpretability.
method Introduces HFNO, a novel FNO-based architecture that processes wavenumber bins in parallel, enhancing interpretability.
result HFNO decomposes turbulent flows across various scales, enabling increased interpretability and multiscale modeling.
New method uses machine learning to estimate sensitivity without binning.
problem Estimating sensitivity of high-dimensional data sets without binning.
method Combines machine-learning classification with likelihood-based inference tests using Kernel Density Estimators.
result Significance estimation is not sensitive to non-smooth probability distributions.
New method for neural networks to predict histogram data.
problem Lack of principled approach for histogram regression.
method Pinball loss applied to cumulative histogram.
result Accuracy similar to EMD with less computational cost.
In this paper we perform a statistical analysis over the returns and relative prices of the CAC and the S\&P with the purpose of analyzing the intra-day seasonalities of single and cross-sectional stock dynamics. In order to do that, we characterized the dynamics of a stock (or a set of stocks) by the evolut…
An Entropy-Based Test and Development Framework for Uncertainty Modeling in Level-Set Visualizationscs.HC
The study evaluates different probability models for uncertainty visualization using entropy calculations.
problem Choosing the right probability model affects memory use, run time, and accuracy in uncertainty visualization.
method Entropy calculation on ensemble data to compare various probability models (uniform, Gaussian, histogram, quantile).
result Models matching the ensemble data distribution have the lowest entropy, indicating better accuracy.
Given discrete time observations over a fixed time interval, we study a nonparametric Bayesian approach to estimation of the volatility coefficient of a stochastic differential equation. We postulate a histogram-type prior on the volatility with piecewise constant realisations on bins forming a partition of the time in…
Paper introduces RPWithPrior for efficient label differential privacy in regression.
problem Protecting user privacy in regression tasks with minimal accuracy loss.
method Modeling responses as continuous random variables, avoiding discretization; estimating optimal intervals for randomized responses.
result RPWithPrior algorithm guarantees ε-label differential privacy and outperforms existing methods.
We present sparse tree-based and list-based density estimation methods for binary/categorical data. Our density estimation models are higher dimensional analogies to variable bin width histograms. In each leaf of the tree (or list), the density is constant, similar to the flat density within the bin of a histogram. His…
For various applications, the relations between the dependent and independent variables are highly nonlinear. Consequently, for large scale complex problems, neural networks and regression trees are commonly preferred over linear models such as Lasso. This work proposes learning the feature nonlinearities by binning fe…
Modeling default contagion and systemic risk using a balls-and-bins approach.
problem Understanding and quantifying systemic risk in financial networks.
method Tractable model, balls-and-bins representation, type space classification, limit theorems.
result Asymptotic Gaussian fluctuations in the final size of default cascades.
PoET-BiN reduces power consumption in neural networks on embedded devices.
problem Power inefficiency in neural network implementations on embedded platforms.
method Look-Up Table based implementation with a modified Decision Tree approach.
result Near state-of-the-art results with up to 6 orders of magnitude energy reduction.
The fast demographic growth, together with the concentration of the population in cities and the increasing amount of daily waste, are factors that push to the limit the ability of waste assimilation by Nature. Therefore, we need technological means to make an optimal management of the waste collection process, which r…