Bayesian Topic Regression models causal inference with text and numerical data.
problem Causal inference using observational text data with both text and numerical confounders.
method Combines supervised Bayesian topic model with Bayesian regression framework, respecting the Frisch-Waugh-Lovell theorem.
result Joint approach recovers ground truth with lower bias than benchmarks, superior prediction results compared to separate approaches.
Proposes a GAN-based model for semi-supervised text regression.
problem Predicting social and economic outcomes from unbalanced textual data.
method Conditional Generative Adversarial Network (GAN) for semi-supervised learning.
result End-to-end framework for predictions without high-level representation selection.
Interpretable text-response modelling for structured outcomes
problem Predicting structured responses alongside textual data
method Joint non-negative matrix factorisation and binomial regression
result Recovering stable response-relevant textual signals
EBMs improve text discrimination by generating negatives from auto-regressive models.
problem Discriminating machine-generated text from human-generated text.
method Use energy-based models to discriminate text, generating negatives using pre-trained auto-regressive language models.
result EBMs can generalize well to changes in generator architectures but are sensitive to training set.
Improved text generation with constraints using discrete auto-regressive biasing.
problem Balancing fluency and constraint satisfaction in LLM outputs.
method Discrete Auto-regressive Biasing, leveraging gradients in discrete text space.
result Significantly improved constraint satisfaction with comparable fluency.
This paper rigorously establishes that the existence of the maximum likelihood estimate (MLE) in high-dimensional logistic regression models with Gaussian covariates undergoes a sharp `phase transition'. We introduce an explicit boundary curve h MLE h_{\text{MLE}} h MLE , parameterized by two scalars measuring the overall magnitu…
XGBoost learns a broad class of functions bridging empirical success and theory.
problem Understanding the function class implicitly learned by XGBoost.
method Introduced an infinite-dimensional function class and a complexity measure, showing optimizers of XGBoost are also optimizers of a penalized regression problem.
result Proved the least squares estimator achieves nearly minimax-optimal convergence rate over the function class.
A new method improves text generation quality and diversity.
problem Exposure bias in Maximum Likelihood Estimation for text generation.
method ψ-MLE, a new training scheme based on density ratio estimation.
result ψ-MLE outperforms Maximum Likelihood Estimation and other models in text generation quality and diversity.
Improved text-conditioned regression using LLMs and diffusion-based neural processes.
problem Major error cascades and computational inefficiency in LLMs for short sequences.
method Combining LLM predictive densities with a diffusion-based neural process.
result Better-calibrated predictions and locally consistent trajectories.
In recent years, stochastic gradient descent (SGD) methods and randomized linear algebra (RLA) algorithms have been applied to many large-scale problems in machine learning and data analysis. We aim to bridge the gap between these two methods in solving constrained overdetermined linear regression problems---e.g., $\el…
A new method estimates expectations from subtractive mixture models without sampling.
problem Estimating expectations from multimodal distributions using SMMs.
method Difference representation of SMMs to create unbiased IS estimator ( Δ e x t E x Δ ext{Ex} Δ e x t E x ). result Demonstrates that Δ e x t E x Δ ext{Ex} Δ e x t E x can achieve comparable estimation quality to auto-regressive sampling but is faster. Improved language models by incorporating statistical discriminators.
problem Distinguishing model-generated text from real text reliably.
method Energy-Based Model framework to incorporate discriminators.
result Improves language model performance in perplexity and human evaluation.
Faster algorithms for Kronecker product regression and low rank approximation.
problem Efficiently solving Kronecker product regression and low rank approximation problems.
method Developed faster algorithms for Kronecker product regression and low rank approximation.
result Significantly faster algorithms for Kronecker product regression and low rank approximation.
Linear Transformer Block combines MLP and linear attention for near-optimal ICL in linear regression.
problem Achieving near-optimal in-context learning (ICL) risk for linear regression with a Gaussian prior.
method Combines linear attention and MLP components in a Linear Transformer Block (LTB). Establishes correspondence with one-step gradient descent estimators ( G D e x t − β \mathsf{GD} ext{-}\mathbfβ GD e x t − β ). result LTB achieves nearly Bayes optimal ICL risk for linear regression with a Gaussian prior.
Proposes RTL model for sentiment classification and key word detection in online reviews.
problem Sentiment classification and key word detection in online reviews for hospitality industry.
method Regularized Text Logistic (RTL) regression model.
result RTL model achieves satisfactory classification performance and identifies key word features.
Paper reduces vocabulary losslessly for language model cooperation.
problem Language models struggle to cooperate with different tokenizations.
method Established a theoretical framework for lossless vocabulary reduction.
result Efficiently converts models with different tokenizations to cooperate with maximal common vocabulary.
Optimizes SGD for text classification using hyper-parameter tuning.
problem Improving performance of SGD in text classification.
method Fine-tuning hyper-parameters using Grid-Search approach.
result Optimized SGD classification leads to better accuracy and execution time.
Improves naturalness in TTS samples using quantized VAE and auto-regressive prosody.
problem Discontinuous and unnatural speech from standard VAE priors.
method Discretized latent features using vector quantization (VQ), and separately trained autoregressive (AR) prior model.
result Significantly improves naturalness in random sample generation.
Compressed LLM embeddings improve noisy regression tasks without overfitting.
problem Noisy regression tasks with high signal-to-noise ratios.
method Comparison of embedding compression techniques using autoencoder hidden representations.
result Compression improves performance on noisy tasks like financial return prediction.
Self-distillation improves model performance in noisy label settings.
problem Improving model accuracy in supervised learning with noisy labels.
method Analyzes self-distillation in two supervised learning problems with noisy labels, using theoretical and empirical approaches.
result Optimal self-distillation parameter is greater than 1 in high label noise regimes, outperforming traditional methods.
Many methods have been used to recognize author personality traits from text, typically combining linguistic feature engineering with shallow learning models, e.g. linear regression or Support Vector Machines. This work uses deep-learning-based models and atomic features of text, the characters, to build hierarchical, …
Proposes a VAE with a discrete bottleneck for better text generation.
problem VAEs struggle with latent variable auto-regressive decoding in text generation.
method Introduces a discretized bottleneck to enforce latent feature matching in a compact space.
result Demonstrates improved text generation capabilities across various tasks.
Benchmarking AutoML for tables with text fields, achieving top performance.
problem Evaluating automated learning systems for tables with text fields.
method Publicly available benchmark with 18 datasets varying in size, types, and feature composition.
result Stack ensembling a multimodal Transformer with various tree models achieved top performance.
Randomly chosen support makes sparse linear regression easy.
problem Sparse linear regression with random support.
method Random support selection for efficient prediction.
result Prediction error ε ε ε with N = e x t p o l y ( k , log d , 1 / ε ) N = ext{poly}(k, \log d, 1/ε) N = e x t p o l y ( k , log d , 1/ ε ) samples and e x t p o l y ( d , N ) ext{poly}(d,N) e x t p o l y ( d , N ) run-time. Autoencoders have been successful in learning meaningful representations from image datasets. However, their performance on text datasets has not been widely studied. Traditional autoencoders tend to learn possibly trivial representations of text documents due to their confounding properties such as high-dimensionality…
Given a matrix A ∈ R n × d \mathbf{A}\in\mathbb{R}^{n\times d} A ∈ R n × d and a vector b ∈ R d b \in\mathbb{R}^{d} b ∈ R d , we show how to compute an ε ε ε -approximate solution to the regression problem min x ∈ R d 1 2 ∥ A x − b ∥ 2 2 \min_{x\in\mathbb{R}^{d}}\frac{1}{2} \|\mathbf{A} x - b\|_{2}^{2} min x ∈ R d 2 1 ∥ A x − b ∥ 2 2 in time O ~ ( ( n + d ⋅ κ sum ) ⋅ s ⋅ log ε − 1 ) \tilde{O} ((n+\sqrt{d\cdotκ_{\text{sum}}})\cdot s\cdot\logε^{-1}) O ~ (( n + d ⋅ κ sum ) ⋅ s ⋅ log ε − 1 ) where …
Paper introduces MLRH, a probabilistic model for multilabel classification.
problem Multilabel classification challenges in various domains.
method Introduces hidden variables to relax one-hot-encoding in logistic regression.
result Probabilistic model achieves competitive performance compared to other algorithms.
Team QCRI-MIT detects hyperpartisan news with 72.9% accuracy.
problem Detecting hyperpartisan news from biased political content.
method Logistic regression model using engineered features from propaganda detection.
result Significant performance improvements with better feature pre-processing.
We introduce a new loss function for evaluating forecasts and estimate models using it.
problem Lack of a decision-theoretic foundation for evaluating forecasts using the Nash-Sutcliffe efficiency.
method We introduce and analyze the Nash-Sutcliffe loss function and its application in estimating models.
result Nash-Sutcliffe loss provides a decision-theoretic foundation for evaluating and estimating models.
FRAPPE estimates tensor canonical rank without CPD computation.
problem Estimating the canonical rank of tensors efficiently.
method Generates synthetic data matching input tensor's size and sparsity, trains a regression model to estimate rank.
result 24 times faster than best baseline, 10% improvement in MAPE on synthetic dataset.
Active learning improves text classification performance by forecasting earlier.
problem Efficiently forecasting text classification model performance with active learning.
method Logarithmic models regressed on active learning data to forecast performance.
result F-measure is more difficult to forecast than accuracy.
Study compares BERT with other sentiment analysis models.
problem Comparing sentiment analysis techniques.
method Used four models: Sent WordNet, logistic regression, LSTM, and BERT on IMDB movie reviews.
result BERT outperformed other models in sentiment classification.
Improved algorithms solve ℓ p \ell_p ℓ p -norm regression problems efficiently.
problem Efficiently solving ℓ p \ell_p ℓ p -norm regression problems for p ∈ ( 1 , 2 ) ∪ ( 2 , ∞ ) p \in (1,2) \cup (2,\infty) p ∈ ( 1 , 2 ) ∪ ( 2 , ∞ ) . method Iterative refinement scheme using smoothed ℓ p \ell_p ℓ p -norms to improve solutions. result Solves ℓ p \ell_p ℓ p -norm regression to 1 / e x t p o l y ( n ) 1 / ext{poly}(n) 1/ e x t p o l y ( n ) accuracy in i l d e O p ( m 1 3 ) ilde{O}_p(m^{\frac{1}{3}}) i l d e O p ( m 3 1 ) iterations. Study finds exposure bias distortion is limited and not incremental in open-ended text generation.
problem Exposure bias in auto-regressive language models causing incremental distortion.
method Proposed metrics to quantify exposure bias impact, used ground-truth prefixes instead of model-generated prefixes.
result Exposure bias distortion is limited and not incremental during generation.
A new LDA model with covariates for mixed-membership clusters.
problem Modeling mixed-membership clusters in discrete data with covariates.
method Negative binomial regression embedded within LDA, slice sampling within Gibbs sampling.
result Model successfully retrieves true parameter values and predicts cluster abundances using covariates.
Guided Flows enhance sample quality in conditional image generation and text-to-speech.
problem Improving sample quality in conditional generative models.
method Integrating classifier-free guidance into Flow Matching (FM) models for Continuous Normalizing Flows (CNFs).
result Guided Flows significantly improve sample quality in conditional image generation and text-to-speech synthesis.
Rejoinder on slope heuristics for model selection in regression.
problem Model selection in least-squares fixed-design regression with biased models and general noise.
method Proves the slope heuristics works even with significant bias and computes expectations for Gaussian noise.
result The slope heuristics is valid even when models are biased and noise has a general dependence structure.
New approach to topic modelling with covariates for large text corpora.
problem Complex topic modelling in large text corpora.
method Combining convex NMF with regression for tractable estimation.
result Faster, interpretable, and better inferential justification than generative models.
We introduce supervised latent Dirichlet allocation (sLDA), a statistical model of labelled documents. The model accommodates a variety of response types. We derive an approximate maximum-likelihood procedure for parameter estimation, which relies on variational methods to handle intractable posterior expectations. Pre…
We consider a sparse linear regression model Y=Xβ^{*}+W where X has a Gaussian entries, W is the noise vector with mean zero Gaussian entries, and β^{*} is a binary vector with support size (sparsity) k. Using a novel conditional second moment method we obtain a tight up to a multiplicative constant approximation of th…
The study introduces a high-dimensional tail index model for viral post analysis.
problem Empirical observation of power-law distributions in viral posts.
method High-dimensional tail index regression model, regularized estimator, debiasing for inference.
result Consistency and asymptotic normality of debiased estimator.
Lipper synthesizes speech from silent videos, improving over single-view methods.
problem Lipreading as text classification is limited; multi-view approach needed.
method Multi-view lipreading as a regression task, producing speech from silent videos.
result Improvement in speech reconstruction with multi-view silent videos.
This paper presents the beginnings of an automatic statistician, focusing on regression problems. Our system explores an open-ended space of statistical models to discover a good explanation of a data set, and then produces a detailed report with figures and natural-language text. Our approach treats unknown regression…
Proposes isotonic regression for calibrating Deep Cox models' survival probabilities.
problem Poor calibration of Deep Cox models' survival probabilities.
method Isotonic regression for post hoc calibration of Deep Cox models.
result Establishes favorable theoretical guarantees and demonstrates empirical effectiveness.
The study explores how machine learning can enhance scientific research.
problem Improving scientific models with machine learning.
method Analysis of data-driven models versus manually added variables in regression.
result Complex models may not always improve over simpler ones in scientific contexts.
Model uses LLMs to process numerical data guided by natural language descriptions.
problem Challenges in integrating prior knowledge into probabilistic models.
method Developed LLM Processes to condition numerical predictive distributions on natural language.
result Improved predictive performance and structured qualitative descriptions.
The abstract reviews models for analyzing count data.
problem Challenges in analyzing count data with standard methods.
method Review of generalized linear models and multinomial models.
result Fundamental connections between multinomial and count models.
Efficiently trains large-scale ordinal regression models using DCD.
problem Efficiently training large-scale ordinal regression models.
method Dual coordinate descent method (DCD) for training and a new prediction function.
result Extensive experiments show the DCD method is suitable for large-scale data.