Data processing inequalities link Fisher information to local differential privacy constraints.
problem Understanding how Fisher information scales with local differential privacy constraints.
method Developed data processing inequalities for Fisher information under local differential privacy.
result Implications for private estimation with optimal bounds and error rates.
Unified framework for deriving generalization bounds in supervised learning.
problem Generalization error bounds in supervised learning.
method Data Processing Inequality PAC-Bayesian framework.
result Unified bounds on binary Kullback-Leibler generalization gap for various divergences.
Study hypothesis testing under quantized samples with communication constraints, achieving near-optimal sample complexity.
problem Optimizing hypothesis testing with quantized samples and communication constraints.
method Developed a polynomial-time algorithm achieving near-optimal sample complexity under communication constraints.
result Achieved near-optimal sample complexity under communication constraints, with a logarithmic factor increase over unconstrained setting.
Paper sets fundamental limits for distributed covariance estimation with constrained communication.
problem Estimating high-dimensional covariance matrices in a feature-split setting with limited communication.
method Developed a Conditional Strong Data Processing Inequality (C-SDPI) to establish minimax lower bounds and an optimal estimation protocol.
result Achieved nearly optimal estimation protocol with sample and communication requirements matching lower bounds up to logarithmic factors.
The data processing inequality doesn't always hold in practice, showing benefits in low-level tasks.
problem The data processing inequality suggests no benefit in pre-processing for classification.
method Theoretical and empirical study of binary classification setup with deep neural networks.
result Pre-classification processing can improve classification accuracy for any finite number of training samples.
We study the tradeoff between the statistical error and communication cost of distributed statistical estimation problems in high dimensions. In the distributed sparse Gaussian mean estimation problem, each of the m machines receives n data points from a d-dimensional Gaussian distribution with unknown mean θ w…
The paper analyzes convergence rates of Langevin dynamics and Proximal Sampler using Φ-divergence.
problem Analyzing convergence rates of Langevin dynamics and Proximal Sampler.
method Extending mixing time analyses to Φ-divergence, using strong data processing inequalities. result Convergence of Φ-divergence to 0 exponentially fast along Unadjusted Langevin Algorithm and Proximal Sampler. New approach shows data memorization trade-offs in large models.
problem Data memorization in large language models and its privacy implications.
method Developed a new approach using strong data processing inequalities to prove lower bounds on memorization.
result Proved that Ω(d) bits of training data information must be memorized for O(1) examples, decaying with example growth. New measures generalize existing ones, linking information and risk.
problem Linking information measures and risk in statistical decision problems.
method Introducing new families of divergence measures and deriving an information processing equality.
result Extension of variational φ-divergence representation to multiple distributions. Quantum states can be learned efficiently using gentle measurements.
problem Efficiently learning quantum states with minimal measurements.
method Introducing α-LGM measurements and proving strong quantum DPI.
result The number of states needed for accurate learning is of order 1/(ε^2 α^2).
Paper outlines a new mathematical language for experiments.
problem Formalizing the scientific process for automation.
method Formulates the scientific process in precise mathematical language.
result Novel contributions in data processing, bias variance, and deficiency.
Locally private methods detect changes in time series data.
problem Detecting distributional changes in time series data under local differential privacy.
method Proposed locally differentially private algorithms based on randomized response and binary mechanisms.
result Theoretical performance bounds and empirical validation of detection accuracy.
Review of information plane analyses in neural networks, highlighting mixed results and methodological challenges.
problem Understanding the relationship between information-theoretic compression and neural network performance.
method Literature review and detailed analysis of information quantity estimation methods.
result Information plane compression is not necessarily information-theoretic but compatible with geometric compression.
A study on the communication complexity of estimating correlations between variables.
problem Estimating the correlation between two sets of correlated random variables with limited communication.
method One-way interactive protocol exchanging k bits, optimizing over interaction protocol and estimator.
result Achieves optimal performance with communication complexity of 1/k, improving over naive schemes.
Recurrent Neural Networks (RNNs) achieve state-of-the-art results in many sequence-to-sequence modeling tasks. However, RNNs are difficult to train and tend to suffer from overfitting. Motivated by the Data Processing Inequality (DPI), we formulate the multi-layered network as a Markov chain, introducing a training met…
Privacy amplification improved through contraction coefficients and Eγ-divergence.
problem Improving privacy guarantees in iterative algorithms.
method Using contraction coefficients derived from Eγ-divergence to determine differential privacy parameters. result Tighter bounds on differential privacy parameters of iterative algorithms.
The matrix-based Renyi's α-entropy functional and its multivariate extension were recently developed in terms of the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the utility and possible applications of these new estimators are rather new an…
New research shows larger language models improve data processing for diverse entries.
problem Optimizing data processing for tables with diverse string entries.
method Analytical tasks on tables with varying language model sizes and a fuzzy join benchmark.
result Larger language models improve data processing for diverse entries, but fine-tuning is necessary.
Study on Mirror Langevin diffusions for convergence rates and Markov chain approximations.
problem Convergence rates and approximations for Mirror Langevin diffusions.
method Lyapunov function methods, Poincaré and log-Sobolev inequalities, Gibbs sampler, Sinkhorn Markov chain.
result Sufficient conditions for exponential convergence of Mirror Langevin diffusions and a guaranteed convergence rate for the Markov chain approximation.
Optimizes real-time data processing in HFT algorithms using machine learning.
problem Optimizing data processing speed in high-frequency trading.
method Adaptive feature selection mechanism, clustering, feature weight analysis, lightweight neural networks.
result The model maintains consistent performance across varying market conditions.
This paper introduces a nonparametric copula-based index for detecting the strength and monotonicity structure of linear and nonlinear statistical dependence between pairs of random variables or stochastic signals. Our index, termed Copula Index for Detecting Dependence and Monotonicity (CIM), satisfies several desirab…
Deep learning applied to SAR data is explored in this paper.
problem Limited use of deep learning in SAR data processing.
method Introduction to relevant deep learning models, analysis of SAR data characteristics, review of state-of-the-art applications, and future research directions.
result Unlocking the potential of deep learning in SAR data processing.
Unified approach to data processing using gauge theory.
problem Data representation and analysis with consistent symmetry.
method Geometric gauge theory for discrete vector bundles.
result Unified understanding of heat kernel properties and data transformation.
The article examines scaling methods and data splitting for financial time series in machine learning.
problem Improving stationarity and information preservation in financial time series data.
method Introduction and comparison of scaling methods, empirical test, and data splitting techniques.
result Proposes effective data splitting methods to prevent overfitting and improve model performance.
The paper studies how quickly samples from Langevin dynamics become independent.
problem Understanding the dependence between samples along Langevin dynamics and related algorithms.
method Measures dependence via Φ-mutual information and proves strong data processing inequalities. result The Φ-mutual information between samples decreases exponentially to zero. We address the problem of prediction of multivariate data process using an underlying graph model. We develop a method that learns a sparse partial correlation graph in a tuning-free and computationally efficient manner. Specifically, the graph structure is learned recursively without the need for cross-validation or p…
Characterizes concept classes for optimistic online learning.
problem Understanding minimal assumptions for online learnability.
method Investigates two questions about concept classes' learnability.
result Characterizes all concept classes for optimistically universal online learnability.
A new framework for information theory considers computational constraints.
problem Understanding information in complex systems with computational limitations.
method Variational extension of Shannon's information theory with computational constraints.
result Predictive V-information can be created through computation and reliably estimated from data. This paper introduces a new property of estimators of the strength of statistical association, which helps characterize how well an estimator will perform in scenarios where dependencies between continuous and discrete random variables need to be rank ordered. The new property, termed the estimator response curve, is e…
Despite their great success in practical applications, there is still a lack of theoretical and systematic methods to analyze deep neural networks. In this paper, we illustrate an advanced information theoretic methodology to understand the dynamics of learning and the design of autoencoders, a special type of deep lea…
RAMANMETRIX simplifies Raman spectroscopy data analysis.
problem Complexity of Raman spectroscopy data processing.
method User-friendly software with chemometric analysis workflow.
result Facilitates integration of Raman spectroscopy into clinical routines.
Paper resolves open problems on sample complexity in binary hypothesis testing.
problem Open problems in distributed simple binary hypothesis testing under information constraints.
method One-shot lower bound on Bayes error, streamlined sample complexity formula, reverse data-processing inequality.
result Optimally tight sample complexity bounds for communication-constrained simple binary hypothesis testing.
Study of symplectic Stiefel and Grassmann manifolds with geodesics and applications.
problem Understanding symplectic bases and subspaces for data processing.
method Lie group approach to derive geodesics and retractions for pseudo-Riemannian and Riemannian metrics.
result Efficient formulas for geodesics and retractions on symplectic manifolds.
AugMix improves model robustness and uncertainty.
problem Improving model performance under distribution shifts.
method AugMix: a simple data processing technique.
result Significantly improves robustness and uncertainty measures.
This paper introduces an information theoretic co-training objective for unsupervised learning. We consider the problem of predicting the future. Rather than predict future sensations (image pixels or sound waves) we predict "hypotheses" to be confirmed by future sensations. More formally, we assume a population distri…
Reshef et al. recently proposed a new statistical measure, the "maximal information coefficient" (MIC), for quantifying arbitrary dependencies between pairs of stochastic quantities. MIC is based on mutual information, a fundamental quantity in information theory that is widely understood to serve this need. MIC, howev…
Functional Magnetic Resonance Imaging (fMRI) relies on multi-step data processing pipelines to accurately determine brain activity; among them, the crucial step of spatial smoothing. These pipelines are commonly suboptimal, given the local optimisation strategy they use, treating each step in isolation. With the advent…
TODS automates time series outlier detection with customizable pipelines.
problem Automated detection of outliers in time series data.
method Modular system with 70 primitives for data processing, time series analysis, and detection algorithms. GUI and data-driven searcher for pipeline design.
result Automated discovery and construction of effective outlier detection pipelines.
This study reviews and evaluates clustering methods for single-cell RNA-seq data.
problem Identifying and characterizing novel cell types from single-cell RNA-seq data.
method Review and performance comparison of clustering methods.
result Performance comparison experiments on two datasets.
Improved diffusion bridge sampling with rKL-LD loss.
problem Improving sampling from unnormalized distributions using diffusion bridges.
method Employing the rKL-LD loss instead of the Log Variance (LV) loss for diffusion bridges.
result rKL-LD consistently outperforms LV loss in diffusion bridges.
Efficiently scheduling data processing jobs on distributed compute clusters requires complex algorithms. Current systems, however, use simple generalized heuristics and ignore workload characteristics, since developing and tuning a scheduling policy for each workload is infeasible. In this paper, we show that modern ma…
All you need is log
problem Comparing multiple probability distributions
method Characterizing multi-distribution Rényi divergences
result Found a canonical multi-distribution Rényi calculus
When using stochastic gradient descent to solve large-scale machine learning problems, a common practice of data processing is to shuffle the training data, partition the data across multiple machines if needed, and then perform several epochs of training on the re-shuffled (either locally or globally) data. The above …
This work introduces a tensor-based method to perform supervised classification on spatiotemporal data processed in an echo state network. Typically when performing supervised classification tasks on data processed in an echo state network, the entire collection of hidden layer node states from the training dataset is …
We study Eγ-divergence contraction and its privacy implications.
problem Analyzing privacy in data processing and algorithms.
method Generalizing Dobrushin's coefficient to Eγ-divergence and deriving contraction coefficients. result Local differential privacy can be expressed in terms of Eγ-divergence contraction, leading to precise sample size reductions. This research optimizes energy consumption forecasting in Puno using parallel computing and ARIMA models.
problem Improving energy consumption forecasting accuracy and efficiency in Puno.
method Parallel computing and ARIMA models for forecasting energy consumption.
result Notable improvements in computational efficiency and data processing capabilities.
A new family of multi-distribution divergences is characterized for fairness and other problems.
problem Comparing more than two probability distributions in a consistent way.
method Characterized a new family of multi-way coincidence divergences.
result The new family of multi-distribution Rényi divergences is necessary and arises from several independent routes.
This research proposes a CL model for RNNs to handle sequential data without forgetting.
problem Learning in dynamic environments without forgetting previous knowledge for sequential data.
method A Recurrent Neural Network (RNN) model with Elastic Weight Consolidation (EWC) for CL.
result The proposed model outperforms EWC and RNNs on CL benchmarks for sequential data.