arXiv research

This work explains how linear representations in large language models arise from training objectives and gradient descent.

A scalable method for Bayesian inference in large linear models.

LoLCATs improves linearized LLM quality with less memory and compute.

Paper proposes a new activation function to reduce overfitting and large weight update issues.

Extends linear representation hypothesis to categorical and hierarchical concepts in LLMs.

The paper analyzes the generalizability of linear autoencoders and multivariate linear regression.

JULIA combines multi-linear and nonlinear models for tensor completion.

Efficient algorithms speed up adversarial training for linear models.

Efficient knockoffs for large-scale feature selection.

We present a scalable and robust Bayesian inference method for linear state space models. The method is applied to demand forecasting in the context of a large e-commerce platform, paying special attention to intermittent and bursty target statistics. Inference is approximated by the Newton-Raphson algorithm, reduced t…

A new method for LDA using randomized Kaczmarz improves accuracy for large datasets.

The paper explores linear representations in language models using counterfactuals.

Auto-Associative models cover a large class of methods used in data analysis. In this paper, we describe the generals properties of these models when the projection component is linear and we propose and test an easy to implement Probabilistic Semi-Linear Auto- Associative model in a Gaussian setting. We show it is a g…

A new method for estimating large-scale linear models with improved precision.

We analyze the (unconditional) distribution of a linear predictor that is constructed after a data-driven model selection step in a linear regression model. First, we derive the exact finite-sample cumulative distribution function (cdf) of the linear predictor, and a simple approximation to this (complicated) cdf. We t…

New method for Bayesian inference on large datasets.

The paper identifies five extreme learning regimes for large linear autoencoders.

Graphical notation simplifies complex polynomial constraints in linear models.

This paper considers generalized linear models using rule-based features, also referred to as rule ensembles, for regression and probabilistic classification. Rules facilitate model interpretation while also capturing nonlinear dependences and interactions. Our problem formulation accordingly trades off rule set comple…

Study high-dimensional Bayesian linear regression using variational inference.

We show how to convert ICL in linearized transformers into model weights.

Study shows linear models can predict CATE without overfitting, even with large data.

Unified framework for studying softmax attention under large prompts.

Optimal downsampling improves GLM performance in imbalanced classification.

Efficiently simulates the Heston model with large time steps using a novel method.

CausalMan simulates complex causality for fair benchmarking.

New insights into how large learning rates affect transformer training dynamics.

Equity auctions show linear price impact up to a large volume, then non-linear.

New method for accurate Bayesian inference in GLMMs for large-scale data.

Local model for Poisson manifolds around submanifolds.

Quadratic models explain neural network behavior during training.

Region-specific linear models are widely used in practical applications because of their non-linear but highly interpretable model representations. One of the key challenges in their use is non-convexity in simultaneous optimization of regions and region-specific models. This paper proposes novel convex region-specific…

New algorithm speeds up fitting GLLVMs to large datasets.

New method reduces SBL complexity from cubic to linear, improving scalability.

CoLA automates efficient numerical linear algebra for complex matrix structures.

Modern statistical applications involving large data sets have focused attention on statistical methodologies which are both efficient computationally and able to deal with the screening of large numbers of different candidate models. Here we consider computationally efficient variational Bayes approaches to inference …

Paper proposes linear transformers for efficient in-context learning without context length limitations.

This paper deals with the problem of large-scale linear supervised learning in settings where a large number of continuous features are available. We propose to combine the well-known trick of one-hot encoding of continuous features with a new penalization called \emph{binarsity}. In each group of binary features comin…

Graph Convolutional Networks (GCNs) are state-of-the-art graph based representation learning models by iteratively stacking multiple layers of convolution aggregation operations and non-linear activation operations. Recently, in Collaborative Filtering (CF) based Recommender Systems (RS), by treating the user-item inte…

CTR prediction in real-world business is a difficult machine learning problem with large scale nonlinear sparse data. In this paper, we introduce an industrial strength solution with model named Large Scale Piece-wise Linear Model (LS-PLM). We formulate the learning problem with $L_1$ and $L_{2,1}$ regularizers, leadin…

We introduce a new embarrassingly parallel parameter learning algorithm for Markov random fields with untied parameters which is efficient for a large class of practical models. Our algorithm parallelizes naturally over cliques and, for graphs of bounded degree, its complexity is linear in the number of cliques. Unlike…

Unified framework for large-scale hypothesis testing with confounders.

Paper addresses online alignment of large language models under uncertain preference feedback.

We consider the problem of personalization of online services from the viewpoint of ad targeting, where we seek to find the best ad categories to be shown to each user, resulting in improved user experience and increased advertisers' revenue. We propose to address this problem as a task of ranking the ad categories dep…

This paper describes the autofeat Python library, which provides scikit-learn style linear regression and classification models with automated feature engineering and selection capabilities. Complex non-linear machine learning models, such as neural networks, are in practice often difficult to train and even harder to …

Large-scale Gaussian process inference has long faced practical challenges due to time and space complexity that is superlinear in dataset size. While sparse variational Gaussian process models are capable of learning from large-scale data, standard strategies for sparsifying the model can prevent the approximation of …

UCoS avoids forward model evaluations in sampling for large-scale linear inverse problems.

Large models are prevalent in modern machine learning scenarios, including deep learning, recommender systems, etc., which can have millions or even billions of parameters. Parallel algorithms have become an essential solution technique to many large-scale machine learning jobs. In this paper, we propose a model parall…