New method uses low logit rank to simplify complex language models.
problem Understanding and learning from modern language models.
method Exploiting the low logit rank structure of language models for efficient learning.
result An efficient algorithm for learning low logit rank models from queries.
Language models exhibit low-rank structure, which can be used for generation.
problem Understanding the low-dimensional structure of large language models.
method Empirical demonstration and theoretical analysis of the approximate rank of language models' logits.
result Language models can generate responses using linear combinations of unrelated prompts.
Proposes a convex model for mixed logit to handle individual heterogeneity.
problem Non-convex optimization in mixed logit models for individual heterogeneity.
method Sparse and low-rank decomposition for convex formulation.
result Convex formulation avoids simulation-based approximation and unstable model interpretation.
A new method for efficient neural network fine-tuning using queryable low-rank update atoms.
problem Rigidity of static low-rank adaptation methods when input and depth-wise computation vary.
method A shared queryable memory of low-rank update atoms, allowing dynamic and context-sensitive adaptation.
result Improves final test performance and training stability compared to standard low-rank adaptation.
A new network model combines features of DCBM, LSM, and β-model, using a cancellation trick for parameter estimation.
problem Challenging parameter fitting in network models.
method Introducing a cancellation trick to resolve parameter fitting issues in the logit-DCBM.
result R-SCORE significantly improves community detection over existing methods.
In applications such as recommendation systems and revenue management, it is important to predict preferences on items that have not been seen by a user or predict outcomes of comparisons among those that have never been compared. A popular discrete choice model of multinomial logit model captures the structure of the …
New algorithm reduces switching costs in multinomial logit bandit problems.
problem Minimizing switching costs in multinomial logit bandit problems.
method Proposed AT-DUCB and FH-DUCB algorithms with low assortment switching costs.
result AT-DUCB and FH-DUCB algorithms achieve almost optimal minimax regret with low switching costs.
PLD distills knowledge using choice-theoretic Plackett-Luce model.
problem Model compression and knowledge transfer between large and small networks.
method PLD uses a weighted list-wise ranking loss based on the Plackett-Luce model.
result PLD achieves consistent gains across diverse architectures and distillation methods.
MANO normalizes logits to estimate test accuracy without labels.
problem Estimating test accuracy of OOD samples without labels.
method Applies Lp norm to normalized logits. result Achieves state-of-the-art performance across various architectures.
LAWN normalizes logits to improve deep network adaptability and generalization.
problem Large logits and weights lead to overfitting in deep networks.
method Logit Attenuating Weight Normalization (LAWN) constrains weight norms in the final sub-network.
result LAWN improves generalization and adaptability of deep networks.
We study the active learning problem of top-k ranking from multi-wise comparisons under the popular multinomial logit model. Our goal is to identify the top-k items with high probability by adaptively querying sets for comparisons and observing the noisy output of the most preferred item from each comparison. To ac…
New method distills cloud models into edge-friendly ones.
problem Cloud-to-edge model compression with limited data exchange.
method Two-step workflow of deprivatization and distillation.
result Outperforms previous state-of-the-art approaches on various benchmarks.
New model improves website ranking by considering user choices as a whole.
problem Optimizing content ordering for user clicks in website design.
method Introduced multinomial logit (MNL) choice model to LTR framework, proposing UCB algorithms.
result Proved theoretical bounds on regret for UCB algorithms in both known and unknown position parameter settings.
With the scale of data growing every day, reducing the dimensionality (a.k.a. sketching) of high-dimensional data has emerged as a task of paramount importance. Relevant issues to address in this context include the sheer volume of data that may consist of categorical samples, the typically streaming format of acquisit…
Logit regularization induces logit clustering, affecting classifier performance.
problem Understanding the mechanism of logit regularization in classification.
method Analysis of logit regularization in linear classification, proving logit clustering leads to Fisher's Linear Discriminant alignment.
result Logit regularization can halve critical sample complexity and induce robust generalization.
Improves adversarial robustness by constraining logits with a bounded function.
problem Improving adversarial robustness in deep learning models.
method Addition of a bounded function before softmax to constrain logits.
result Our method improves adversarial robustness without requiring adversarial training.
Logit-GFN accelerates GFlowNets training by scaling logits based on temperature.
problem Training temperature-conditional GFlowNets is numerically challenging.
method Logit-GFN uses a learned function of temperature to scale policy logits.
result Logit-GFN greatly accelerates GFlowNets training and improves generalization and mode discovery.
A new logit model derived from the Weibull manifold.
problem No potential function on the Weibull manifold.
method Extracted a logit model from the two-parameter Weibull model.
result Found a completely integrable Hamiltonian gradient system on the logit model.
Proposes SOVR loss to improve adversarial robustness by increasing logit margins.
problem Adversarial training's difficulty in robustness against sophisticated attacks.
method Introduces SOVR loss function that switches from cross-entropy to one-vs-the-rest loss for important samples.
result SOVR loss increases logit margins of important samples, improving robustness against Auto-Attack.
In this paper, we develop improved techniques for defending against adversarial examples at scale. First, we implement the state of the art version of adversarial training at unprecedented scale on ImageNet and investigate whether it remains effective in this setting - an important open scientific question (Athalye et …
Ranking data arises in a wide variety of application areas but remains difficult to model, learn from, and predict. Datasets often exhibit multimodality, intransitivity, or incomplete rankings---particularly when generated by humans---yet popular probabilistic models are often too rigid to capture such complexities. In…
CRS model improves ranking data modeling with theoretical guarantees.
problem Lack of rich, multimodal models for ranking data.
method Contextual Repeated Selection (CRS) model for multimodal ranking data.
result CRS model significantly outperforms existing methods in various ranking contexts.
Generative classifier derived from any discriminative classifier rejects illegal inputs.
problem Detecting and rejecting illegal inputs like adversarial examples and out-of-distribution samples.
method SDIM-logit: learns generative classifier from logits of any discriminative classifier, imposing statistical constraints.
result SDIM-logit inherits performance of base classifier without loss and can reject illegal inputs.
This paper explains how low-precision arithmetic causes loss spikes in deep learning models.
problem Loss spikes during long-term training of deep neural networks.
method Analyzes the impact of floating-point precision limits on gradient updates and feature means.
result Numerical Feature Inflation (NFI) explains loss spikes and rapid parameter norm growth.
RCPO uses ranked choice modeling for better LLM alignment.
problem Pairwise preference optimization limits LLM alignment.
method Unified framework combining preference optimization and ranked choice modeling.
result RCPO outperforms competitive baselines in LLM alignment.
Logit dynamics formula reveals self-regulation in softmax policy gradient methods.
problem Understanding the stability and convergence of softmax policy gradient methods.
method Deriving the exact formula for the L2 norm of the logit update vector.
result Logit update magnitudes are modulated by action probability and policy concentration.
Recently, Kannan et al. [2018] proposed several logit regularization methods to improve the adversarial robustness of classifiers. We show that the computationally fast methods they propose - Clean Logit Pairing (CLP) and Logit Squeezing (LSQ) - just make the gradient-based optimization problem of crafting adversarial …
In many real-world prediction tasks, class labels include information about the relative ordering between labels, which is not captured by commonly-used loss functions such as multi-category cross-entropy. Recently, the deep learning community adopted ordinal regression frameworks to take such ordering information into…
Logit models are usually applied when studying individual travel behavior, i.e., to predict travel mode choice and to gain behavioral insights on traveler preferences. Recently, some studies have applied machine learning to model travel mode choice and reported higher out-of-sample predictive accuracy than traditional …
Logit distance bounds representational similarity of models.
problem Approximating linear similarity when distributions are close.
method Defined a logit distance and proved its relationship to representational dissimilarity.
result Logit distance bounds representational similarity, providing nontrivial control in practice.
Rank-one measurements limit feasible sets for low-rank PSD matrices.
problem Feasibility of PSD matrices under rank-one measurements.
method Characterization of feasible sets for PSD matrices given rank-one projections.
result Radius of feasible sets determines singleton solution sets for low-rank matrices.
FedLoRU improves FL efficiency by using low-rank updates.
problem Communication inefficiency and performance reduction in Federated Learning.
method Proposes FedLoRU, a low-rank update framework for FL, which reduces communication costs while maintaining performance.
result FedLoRU achieves convergence rates similar to FedAvg and is robust to heterogeneous and large numbers of clients.
Low-rank modeling generally refers to a class of methods that solve problems by representing variables of interest as low-rank matrices. It has achieved great success in various fields including computer vision, data mining, signal processing and bioinformatics. Recently, much progress has been made in theories, algori…
New method solves nonsmooth low-rank matrix optimization problems efficiently.
problem Nonsmooth and low-rank matrix optimization problems in statistics and machine learning.
method Low-rank Extragradient Method with warm-start initialization.
result The extragradient method converges to an optimal solution with rate O(1/t) and requires only two low-rank SVDs per iteration. SGD with mini-batches can solve convex low-rank matrix problems efficiently.
problem Solving large-scale convex low-rank matrix problems efficiently.
method Stochastic Gradient Descent with mini-batches and low-rank projections.
result SGD with mini-batches produces low-rank iterates with high probability.
New spectral methods improve matrix estimation in RL with low-rank structure.
problem Estimating matrices with low-rank structure in reinforcement learning.
method Spectral-based matrix estimation approaches.
result Spectral methods efficiently recover singular subspaces and minimize entry-wise error.
Proposes tensor Q-rank for better tensor rank recovery in complex data.
problem Improving tensor rank recovery for complex data with low sampling rate.
method Introduces tensor Q-rank and two selection methods for Q, proposing VMTQN and MOTQN models. result Demonstrates superior performance in tensor completion problems compared to TNN-based methods.
The paper proposes using low rank assumption to improve causal structure learning in DAGs.
problem Challenges in learning causal structures in high-dimensional, non-sparse DAGs.
method Exploits low rank assumption of DAG adjacency matrix to adapt causal structure learning methods.
result Maximum rank is highly related to hubs, suggesting low rank for scale-free networks.
The paper analyzes how low-rank layers in neural networks improve generalization.
problem Understanding how low-rank layers affect generalization in neural networks.
method Applying Maurer's chain rule for Gaussian complexity to analyze rank and spectral norm constraints.
result Deep networks with low-rank layers achieve better generalization than those with full-rank layers.
Novel method for efficient low-rank matrix estimation and bandit algorithms.
problem Low-rank matrix estimation and bandit problems.
method LowPopArt method for low-rank matrix estimation and novel experimental design criterion.
result Improved recovery guarantees and regret bounds for low-rank bandit algorithms.
Algorithm recovers multiple low-rank matrices from unlabeled data.
problem Learning mixtures of low-rank models from unlabelled data.
method Three-stage meta-algorithm that copes with non-convexity and noise.
result Near-optimal sample and computational complexities under Gaussian designs.
In this paper, we consider the problem of low-rank phase retrieval whose objective is to estimate a complex low-rank matrix from magnitude-only measurements. We propose a hierarchical prior model for low-rank phase retrieval, in which a Gaussian-Wishart hierarchical prior is placed on the underlying low-rank matrix to …
Paper improves MVSC using tensor low-rank modeling.
problem Improving multi-view spectral clustering.
method Structured tensor low-rank norm for MVSC optimization.
result Proposed method outperforms state-of-the-art methods.
Derives smooth homogeneous structures for low-rank tensors.
problem Understanding the geometry of low-rank tensors.
method Analyzes sets of fixed CP, multilinear, and TT rank tensors to derive smooth homogeneous manifolds.
result Derives Riemannian metrics with complete geodesics.
We consider neural network training, in applications in which there are many possible classes, but at test-time, the task is a binary classification task of determining whether the given example belongs to a specific class, where the class of interest can be different each time the classifier is applied. For instance, …
Bayesian model improves image completion accuracy by automatically learning low rank structure.
problem Improving image completion accuracy with limited data and avoiding overfitting.
method Developed a Bayesian low rank tensor ring model with multiplicative interaction and Student-T distribution for sparse core factors.
result The proposed method outperforms state-of-the-art image completion techniques, especially in recovery accuracy.
Low-rank approximation is an effective model compression technique to not only reduce parameter storage requirements, but to also reduce computations. For convolutional neural networks (CNNs), however, well-known low-rank approximation methods, such as Tucker or CP decomposition, result in degraded model accuracy becau…
Low-rank structure emerges in neural networks during learning.
problem Understanding the evolution of synaptic connectivity over learning.
method Investigated the rank of 3-tensor formed by weight matrices throughout learning.
result Inferred weights are low-tensor-rank and evolve in a fixed low-dimensional subspace.