The paper offers methods to estimate and infer the boundary of a set-identified linear model.
problem Estimating and inferring the boundary of a set-identified linear model with many covariates.
method The paper uses semiparametric moment equations and Neyman-orthogonality combined with sample splitting to construct a root-N consistent, uniformly asymptotically Gaussian estimator and a multiplier bootstrap procedure for inference.
result The paper provides a method to estimate and infer the boundary of a set-identified linear model.
Proposes using DII to identify non-linear causal relationships in EU Allowances returns.
problem Identifying causal relationships in non-linear data of EU Allowances returns.
method Uses Differentiable Information Imbalance (DII) for non-parametric causal discovery compared to multivariate Granger causality.
result Significant overlap and differences in causal variables identified by linear and non-linear methods.
Extends sublinear expectations to random sets, identifying extremal and constructing methods.
problem Extending sublinear expectations to random sets.
method Identifying extremal expectations and presenting general construction methods.
result Identification of extremal sublinear and superlinear expectations.
We study the singular locus of solutions to Hamilton-Jacobi equations with a Hamiltonian independent of u. In a previous paper, we proved that the singular locus is what we call a balanced split locus. In this paper, we find and classify all balanced split sets, identifying the cases where the only balanced split loc…
A families index theorem in K-theory is given for the setting of Atiyah, Patodi and Singer of a family of Dirac operators with spectral boundary condition. This result is deduced from such a K-theory index theorem for the calculus of cusp, or more generally fibred cusp, pseudodifferential operators on the fibres (with …
Compact set identifies projective special real manifolds.
problem Characterizing projective special real manifolds.
method Constructing a convex generating set Cn. result Closed connected projective special real manifolds correspond to inner points of Cn. We determine the PSL_2(C) and SL_2(C) character varieties of the once-punctured torus bundles with tunnel number one, i.e. the once-punctured torus bundles that arise from filling one boundary component of the Whitehead link exterior. In particular, we determine `natural' models for these algebraic sets, identify them …
Framework identifies population quantities from MNAR feedback using weak shadow variables from pretrained models.
problem Estimating mean outcomes from MNAR user feedback with bias and lack of identification.
method Develops a partial identification framework using linear programs and weak shadow variables from pretrained models.
result Bounds on estimand are obtained by solving linear programs incorporating pretrained model predictions.
This work uncovers algorithm-dependent regularisation in diffusion models.
problem Understanding and improving generalisation in high-dimensional diffusion models.
method Algorithmic stability and score stability analysis.
result Identifies multiple sources of implicit regularisation unique to diffusion models.
Framework achieves fairness in predictions using partially known causal graph over clusters of variables.
problem Achieving fairness in algorithmic decisions when causal graph knowledge is limited.
method Leverages a causal graph over clusters of variables to train a prediction model, reducing interventional distribution discrepancies.
result Framework strikes a better balance between fairness and accuracy than existing approaches under limited causal graph knowledge.
The study reveals fundamental limits of fraud detection in card payment networks.
problem Fraud detection in card payment networks is challenging due to structural information impairments.
method Formalized card authorization as a sequential decision problem with delayed feedback, derived minimax regret lower bound.
result Improving issuer reporting quality or reducing censorship can yield larger reductions in the regret floor than increasing model complexity.
Novel method transfers orometric measures to metric data sets, identifying key items.
problem Identifying key items in metric data sets like knowledge graphs.
method Transfers orometric measures to bounded metric spaces, using 'isolation' and 'prominence' functions.
result Identifies structurally relevant items in geographic data sets of Germany and France.
Robust kNN classifier achieves optimal rates in noisy classification.
problem Classification with unknown asymmetric label noise.
method Robust kNN classifier with additional assumptions.
result Achieves minimax optimal rates in noisy classification.
AWARE improves graph prediction by aggregating walks with attention schemes.
problem Improving graph prediction accuracy using walk aggregation.
method Integrates attention schemes into walk-aggregating GNNs.
result AWARE outperforms existing methods in graph-level prediction tasks.
Analyzes neural networks using linear models to understand their behavior.
problem Understanding multi-layer neural networks through linear models.
method Recalls and reviews four models: linear regression with concentrated features, kernel ridge regression, random feature model, and neural tangent model.
result Highlights limitations of linear theory and discusses approaches to overcome them.
Unified derivation of high-dimensional linear models using stochastic gradient descent.
problem Performance analysis of high-dimensional linear models trained with stochastic gradient descent.
method Derivation of a deterministic equivalence for the two-point function of a random matrix resolvent.
result Unified understanding of model performance including previously known and novel results.
Region-specific linear models are widely used in practical applications because of their non-linear but highly interpretable model representations. One of the key challenges in their use is non-convexity in simultaneous optimization of regions and region-specific models. This paper proposes novel convex region-specific…
BELIEF framework interprets GLMs using binary linear models.
problem Understanding and interpreting generalized linear models (GLMs) with binary outcomes.
method Developed a framework called binary expansion linear effect (BELIEF) to interpret GLMs through transparent linear models.
result BELIEF framework reveals perfect predictors in complete separation scenarios.
Gaussian processes retain the linear model either as a special case, or in the limit. We show how this relationship can be exploited when the data are at least partially linear. However from the perspective of the Bayesian posterior, the Gaussian processes which encode the linear model either have probability of nearly…
PILOT is a fast algorithm for linear model trees that outperforms existing methods.
problem Fitting linear model trees to large datasets efficiently and accurately.
method Greedy training with L2 boosting and model selection rule. result PILOT outperforms standard decision trees and other linear model trees on various datasets.
SLIMs hybridize sparse linear and isotonic models for high-dimensional data.
problem High-dimensional data with non-linear relationships.
method Hybridizing sparse linear models and isotonic models, proposing a two-step algorithm for estimation.
result The algorithm accurately estimates sparse parameters and monotone functions.
Paper presents a machine learning method to improve significance tests for misspecified linear models.
problem Misspecification of linear assumptions in social science models leads to inaccurate significance levels.
method Apply machine learning to fit ground truth function, calculate linear approximation, and adjust the estimator.
result The method significantly outperforms linear regression for non-linear ground truth functions.
Jump Markov linear models consists of a finite number of linear state space models and a discrete variable encoding the jumps (or switches) between the different linear models. Identifying jump Markov linear models makes for a challenging problem lacking an analytical solution. We derive a new expectation maximization …
Analyzes generalization error in generalized linear models, explaining double descent phenomenon.
problem Understanding generalization of machine learning models in high dimensions.
method Develops a framework to characterize asymptotic generalization error for generalized linear models.
result Rigorously explains the double descent phenomenon in generalized linear models.
RGAM builds more accurate models by preferring linear features over non-linear ones.
problem Building accurate models when linearity assumption is poor.
method Multi-stage algorithm guided by the principle of preferring linear features.
result RGAM can fit sparse generalized additive models at scale for various data types.
New approach improves linear-time attention for language models.
problem Challenges of quadratic attention in long-sequence modelling, especially for discrete data.
method Reinterpreting linear attention through latent probabilistic graphical models, introducing asymmetric structure and recurrent parameterisation.
result Our model achieves competitive performance and outperforms existing linear attention variants on language modelling benchmarks.
Linear properties are either universal or absent across language models.
problem Explaining the prevalence of linear properties in language models.
method Proved identifiability of distribution-equivalent next-token predictors and analyzed various notions of linearity.
result Linear properties either hold in all or none distribution-equivalent next-token predictors.
LoLCATs improves linearized LLM quality with less memory and compute.
problem Linearizing large language models (LLMs) often degrades model quality and requires expensive training.
method Two-step method: attention transfer and low-rank adaptation.
result Significant improvement in linearizing quality with 20+ points on 5-shot MMLU.
LQF linearizes deep models for better interpretability.
problem Lack of interpretability in deep neural networks.
method Simple modifications to architecture, loss function, and optimization.
result Comparable performance to non-linear fine-tuning, with interpretability.
Improved electricity price forecasting model combining linear and non-linear structures.
problem Day-ahead electricity price forecasting in energy systems.
method Recurrent neural networks with embedded linear structures.
result Approximately 11% higher accuracy than state-of-the-art models.
Quadratic memory needed for linear prediction in streaming model.
problem Scalable memory-efficient linear prediction in streaming model.
method Memory lower bound for finding orthogonal vectors and estimates on Grassmannian packing.
result Problems cannot be solved by scalable memory-efficient streaming algorithms.
This paper proposes a distributed Bayesian method for piecewise sparse linear models.
problem High computational cost in simultaneous model selection for piecewise linear models.
method Distributed factorized asymptotic Bayesian (FAB) inference on distributed memory architectures.
result Achieves high prediction accuracy and performance scalability.
Improved few-shot learning with interpretable models.
problem Few-shot learning with limited data.
method Linear Distillation Learning using linear functions for each class.
result Better performance compared to other interpretable models.
Efficient algorithm for conditional linear regression.
problem Building models that accurately predict a specific segment of data.
method Efficient algorithm for identifying a significant segment of the population and its linear regression fit.
result Theoretical analysis and efficient algorithm for conditional linear regression.
Python library automates feature engineering and selection for linear models.
problem Difficulties in training and explaining complex machine learning models.
method Automated feature engineering and selection for linear models.
result Improves prediction accuracy of linear models while retaining interpretability.
Review of privacy-preserving linear models for high-dimensional data.
problem Overfitting and data memorization in high-dimensional linear models.
method Comprehensive comparison of optimization techniques for differentially private high-dimensional linear models.
result Coordinate-optimized algorithms perform best in empirical tests.
StarNet trains deep models without gradients using linear equations.
problem Training deep generative models with gradients.
method Solving determined systems of linear equations.
result Least-square bounds for latent codes and model parameters.
This work explains how linear representations in large language models arise from training objectives and gradient descent.
problem Understanding the origins of linear representations in large language models.
method A latent variable model to abstract and formalize concept dynamics, combined with analysis of the softmax cross-entropy objective and gradient descent.
result Linear representations emerge when learning from data matching the latent variable model, and this simple structure suffices to yield linear representations.
Proposes a deep learning model to improve stock market prediction.
problem Lack of interpretability in linear multi-factor models for stock prediction.
method Extends linear multi-factor model to LSTM+LRP for non-linear and time-varying predictions.
result Deep recurrent factor model outperforms traditional models in predictive capability.
We learn linear models from nonlinear systems using multiple trajectories and regularization.
problem Identifying linear models from data when the underlying dynamics are nonlinear.
method Multiple trajectories data acquisition followed by regularized least squares.
result Learn linearized dynamics with arbitrarily small error given enough samples.
Optimizes explanations for linear models to enhance trust and interpretability.
problem Lack of well-defined interpretability for linear models and ad hoc approaches.
method Optimization framework using coordinate updates on coefficients.
result Derives a family of interpretability metrics for linear models.
Develops fast approximations for conditional Shapley values in linear and polynomial models.
problem Estimating conditional Shapley values using regression models is computationally expensive.
method A new approximative estimation method for conditional Shapley values using linear and polynomial regression models.
result Our method significantly reduces computation time compared to existing methods.
The paper analyzes the generalizability of linear autoencoders and multivariate linear regression.
problem Limited theoretical understanding of linear autoencoders' performance.
method Proposes a PAC-Bayes bound for multivariate linear regression and shows LAEs as constrained models.
result The proposed PAC-Bayes bound is tight and correlates with practical metrics.
Log-linear models are the popular workhorses of analyzing contingency tables. A log-linear parameterization of an interaction model can be more expressive than a direct parameterization based on probabilities, leading to a powerful way of defining restrictions derived from marginal, conditional and context-specific ind…
In this paper we investigate general linear stochastic volatility models with correlated Brownian noises. In such models the asset price satisfies a linear SDE with coefficient of linearity being the volatility process. This class contains among others Black-Scholes model, a log-normal stochastic volatility model and H…
Contrastive learning helps linear models understand document topics.
problem Document classification with limited labeled data.
method Contrastive learning applied to document topic modeling.
result Linear models can recover topic posterior information from contrastive learning representations.
A new decision tree variant improves linear model performance.
problem Improving decision tree performance on non-linear data.
method Extremely random tree with non-linear data transformation and linear observer.
result Outperforms linear models on benchmark dataset.
Dynamic linear models improve travel time prediction for congested freeways.
problem Accurate travel time prediction for congested freeways.
method Dynamic linear models (DLMs) with time-varying parameters.
result Significant improvements in travel time prediction accuracy, especially for short-term predictions.