Study finds exposure bias distortion is limited and not incremental in open-ended text generation.
problem Exposure bias in auto-regressive language models causing incremental distortion.
method Proposed metrics to quantify exposure bias impact, used ground-truth prefixes instead of model-generated prefixes.
result Exposure bias distortion is limited and not incremental during generation.
A new robust prefix-tuning framework improves model robustness against adversarial attacks.
problem Lack of robustness in prefix-tuning for adversarial attacks.
method Leveraging layerwise activations of pretrained models for additional prefix finetuning during the test phase.
result Framework substantially improves robustness over strong baselines while maintaining comparable accuracy on clean texts.
CROP verifies clean prefixes in reasoning traces, improving downstream repair accuracy.
problem Uncertainty in reasoning traces prevents full certification of entire responses.
method CROP selects a calibrated threshold to certify the longest prefix with low risk proxies.
result CROP improves downstream repair accuracy by preserving valid reasoning and discarding misleading suffixes.
Prefix consistency improves model reliability by weighting answers based on their reproducibility.
problem Improving the reliability of large language models' reasoning traces.
method Use prefix consistency to weight candidate answers based on their reproducibility during regeneration.
result Prefix consistency is the best correctness predictor, reaching Standard MV plateau accuracy with up to 21x fewer tokens.
Improved text generation with discourse-aware soft prompts.
problem Efficient fine-tuning methods don't generalize across all generation tasks.
method Hierarchical blocking and attention sparsity on prefix parameters.
result Structured design of prefix parameters yields more coherent generations.
ReOPD uses pre-collected teacher trajectories to distill knowledge from multi-turn interactions.
problem The cost of fully online on-policy distillation for multi-turn interactions.
method ReOPD, an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes, addressing the prefix trap and distribution shift.
result ReOPD preserves or improves OPD-level accuracy, uses zero tool calls, and is at least 4imes faster per training step. KVCOMM optimizes multi-agent LLM systems by reusing KV-caches, reducing redundant processing.
problem Substantial overhead from reprocessing overlapping contexts across multi-agent systems.
method KVCOMM reuses KV-caches and adjusts offsets for shared content using a pool of cached examples (anchors).
result Achieves over 70% reuse rate across diverse multi-agent tasks, up to 7.8x speedup.
DKMD is a fast signed statistic for comparing univariate distributions.
problem Comparing univariate distributions, especially preserving directionality.
method DKMD integrates kernel mean embeddings against an odd weighting function.
result DKMD preserves directionality and is robust to outliers.
Autostackability for finitely generated groups is defined via a topological property of the associated Cayley graph which can be encoded in a finite state automaton. Autostackable groups have solvable word problem and an effective inductive procedure for constructing van Kampen diagrams with respect to a canonical fini…
Study risk-constrained Kelly optimization for mutually exclusive outcomes, proving support invariance and developing a structured algorithm.
problem Risk-constrained Kelly optimization for mutually exclusive outcomes with explicit state prices.
method Analyzes the finite mutually exclusive outcome version of risk-constrained Kelly optimization with explicit state prices, proving support invariance and developing a structured algorithm.
result Support is invariant across CRRA parameter and drawdown-surrogate parameter in the overround regime.
This paper explains how to optimize prompts for model adaptation.
problem Understanding and optimizing prompt tuning for model adaptation.
method Bayesian view and meta-learning to explain prompt optimization.
result Optimal prompting can be studied formally as conditioning Bayesian predictors.
New algorithm reduces regret in private online learning with optimal gap-dependent rate.
problem Optimal gap-dependent regret rate for private stochastic decision-theoretic online learning.
method Horizon-free pure-DP algorithm with exponential block partitioning and softmax selection.
result Explicit regret bound of 1000⋅(ΔminlogK+εlogK). Improved bounds on acylindricity for right-angled Artin groups.
problem Bounding the acylindrical action of right-angled Artin groups on their extension graphs.
method Exploring lattice properties, studying prefixes of powers, and extending quasi-root uniqueness.
result Cardinality of r-quasi-stabilizer is bounded by a linear function of r. A new method improves language model extrapolation without changing training.
problem Improving language model performance on sequences not seen during training.
method Perturbation-based training procedure for language models.
result The method consistently improves out-of-support prediction.
We consider the problem of path inference: given a path prefix, i.e., a partially observed sequence of nodes in a graph, we want to predict which nodes are in the missing suffix. In particular, we focus on natural paths occurring as a by-product of the interaction of an agent with a network---a driver on the transporta…
We improve private training accuracy with learning rate schedules and matrix factorizations.
problem Private training with learning rate schedules and correlated noise.
method General upper and lower bounds for learning rate schedules, memory-efficient constructions, and schedule-aware factorizations.
result Schedule-aware factorizations improve accuracy in private training.
The paper identifies patterns in language model weights used for memorizing paragraphs.
problem Locating the specific mechanisms and weights used by language models to memorize paragraphs.
method Examined gradients and attention patterns in language models to identify memorized paragraphs.
result Gradients of memorized paragraphs have a distinguishable spatial pattern, and localized attention heads are involved in paragraph memorization.
Improved math problem solvers using Transformer networks and diverse notations.
problem Challenges in constructing accurate and automatic solvers for math word problems.
method Transformer networks trained to translate math word problems to arithmetic expressions in infix, prefix, and postfix notations. Pre-training on general text corpus to improve performance.
result Significant improvements in accuracy, up to 10% over previous state of the art.
The following discourse is inspired by the works on hyperbolic groups of Epstein, and Neumann/Reeves. Epstein showed that geometrically finite hyperbolic groups are biautomatic. Neumann/Reeves showed that virtually central extensions of word hyperbolic groups are biautomatic. We prove the following generalisation: Theo…
Stage-based hyper-parameter optimization reduces GPU-hours and training time.
problem Efficiently executing hyper-parameter optimization for deep learning models.
method Stage-based execution strategy to remove redundant computations.
result Stage-based execution outperforms trial-based method by up to 6.60 times in GPU-hours and 4.13 times in training time.
Optimizes kernel discrepancies by selecting subsets efficiently.
problem Improving kernel discrepancies for QMC methods.
method Introduces a novel subset selection algorithm for kernel discrepancies.
result Efficiently generates low-discrepancy samples from various distributions.
New discrepancy function compares discrete probability measures considering space geometry.
problem Comparing discrete probability measures in a geometrically meaningful way.
method Proposes the Fourier Discrepancy Function, proving convexity, differentiability, and providing gradient formula.
result Proves the Fourier Discrepancy is convex, twice differentiable, and provides an explicit gradient formula.
This paper introduces localized discrepancy theories for unsupervised domain adaptation.
problem Improving generalization bounds for unsupervised domain adaptation.
method Localized discrepancies defined on the hypothesis space after localization, leading to smaller and asymmetric values.
result Improved generalization bounds and sample complexity reduction.
The article introduces practical estimators for kernel discrepancies.
problem Estimating kernel discrepancies accurately and efficiently.
method Presented various estimators for MMD, HSIC, and KSD, including V-statistics, U-statistics, and incomplete U-statistics. Stressed the importance of kernel bandwidth and introduced adaptive estimators.
result Adaptive estimators combining multiple estimators with various kernels address the problem of kernel selection.
Improved financial VA intent classification accuracy.
problem Determining user intents for unseen open intents.
method Supervised pre-training of intent representations using prefix-tuning and fine-tuning.
result 1.63% - 2.07% higher accuracy on banking77 benchmark.
Paper defines class discrepancy for machine learning problems and provides coresets.
problem Addressing discrepancies in machine learning models.
method Defines class discrepancy, provides techniques for bounding discrepancy, and develops coresets and streaming sketches.
result Establishes coresets of size O(sqrt{d}/epsilon) for various machine learning problems.
Feature noise causes loss discrepancies across groups even with equal data.
problem Loss discrepancies observed in learning procedures across different groups.
method Characterized the effect of feature noise on loss discrepancy in linear regression.
result Feature noise leads to loss discrepancy even when groups have equal data.
MPMC generates low-discrepancy points using graph neural networks.
problem Generating efficient low-discrepancy point sets.
method Leveraging Graph Neural Networks to model geometric properties.
result Achieves state-of-the-art performance in generating low-discrepancy points.
Sliced kernelized Stein discrepancy improves goodness-of-fit tests and model learning in high dimensions.
problem The curse-of-dimensionality in kernelized Stein discrepancy (KSD).
method Sliced Stein discrepancy and its scalable variants using optimal one-dimensional projections.
result Significantly outperforms KSD and baselines in goodness-of-fit tests and improves model learning.
For the numerical solution of the American option valuation problem, we provide a script written in MATLAB implementing an explicit finite difference scheme. Our main contribute is the definition of a posteriori error estimator for the American options pricing which is based on Richardson's extrapolation theory. This e…
Much of machine learning relies on comparing distributions with discrepancy measures. Stein's method creates discrepancy measures between two distributions that require only the unnormalized density of one and samples from the other. Stein discrepancies can be combined with kernels to define kernelized Stein discrepanc…
Braids can be represented geometrically as laminations of punctured disks. The geometric complexity of a braid is the minimal complexity of a lamination that represents it, and tight laminations are representatives of minimal complexity. These laminations give rise to a normal form of braids, via a relaxation algorithm…
Study shows the corrected Akaike criterion is inadmissible for estimating Kullback-Leibler discrepancy.
problem Inadmissibility of the corrected Akaike information criterion for estimating Kullback-Leibler discrepancy.
method Loss estimation framework to demonstrate inadmissibility and provide improved estimators.
result Improved estimators of Kullback-Leibler discrepancy are provided and perform well in reduced-rank situations.
PPT optimizes transformer behavior by steering its latent posterior using prior samples.
problem Eliciting desired behavior from transformers without backpropagation.
method Posterior Prefix Tuning (PPT) uses predictive Monte Carlo (PMC) samples and importance sampling to optimize the latent posterior.
result PPT optimizes transformer behavior without backpropagation, achieving high utility across different utility functions.
Semi-parametric framework for nonlinear system identification
problem Nonlinear system identification
method Orthogonal Gaussian process regression
result Interpretable models from incomplete physics
TMDA aligns subdomain data distribution discrepancies across domains using manifold representations.
problem Transfer learning challenges due to domain divergence.
method TMDA uses low-dimensional manifolds to represent subdomains and aligns local data distribution discrepancies across domains using M3D.
result TMDA is a promising method for various transfer learning tasks.
New partition designs reduce star discrepancy in high-dimensional sampling.
problem Improving the expected star discrepancy in high-dimensional sampling.
method Developed non-equal volume partitions to achieve lower expected star discrepancy.
result Explicit upper bounds for expected star discrepancy under non-equal volume partitions.
Power-SMC reduces inference latency for training-free LLM reasoning.
problem Training-free LLM reasoning with low latency.
method Power-SMC, a training-free Sequential Monte Carlo scheme targeting sequence-level power distribution.
result Power-SMC reduces inference latency from 16-28× to 1.4-3.3× over baseline decoding.
This work proves the optimal estimation rates for popular kernel discrepancies.
problem Estimating the disagreement of distributions using kernel discrepancies.
method Proving minimax lower bounds for MMD, HSIC, and KSD.
result The minimax lower bound for estimation of MMD, HSIC, and KSD is \( n^{-1/2} \) on general topological spaces.
Training-free method improves large language model sequence quality via reward-guided sampling.
problem Optimizing large language model sequence quality over token likelihood.
method Reward-augmented target distribution combined with Sequential Monte Carlo sampling.
result Significant gains in sequence generation and mathematical reasoning tasks.
The paper highlights the importance of model discrepancy in cardiac simulations.
problem Uncertainty in model structure and equations affects predictions.
method The authors use Gaussian processes and autoregressive-moving-average models to account for model discrepancy.
result Different methods to account for model discrepancy have advantages and shortcomings.
A new metric DJP-MMD improves domain adaptation by balancing transferability and discriminability.
problem Improving domain adaptation performance by balancing transferability and discriminability.
method Discriminative Joint Probability Maximum Mean Discrepancy (DJP-MMD) replaces the traditional joint MMD.
result DJP-MMD outperforms traditional MMDs in image classification tasks.
A new method for density estimation using mixture discrepancy and moments.
problem Generalizing histogram statistics to higher dimensions.
method Density estimation via mixture discrepancy and moments (DSP-mix and MSP).
result DSP-mix and MSP are computationally tractable and maintain accuracy with increased speed.
Stochastic Stein Discrepancies improve inference efficiency.
problem Intractable computation of Stein discrepancies.
method Subsampled approximations of Stein operators.
result Stochastic Stein Discrepancies inherit convergence properties of standard SDs.
Bayes-consistent disagreement discrepancy loss improves model robustness.
problem Distribution shift in real-world neural network deployment.
method Introducing a novel disagreement loss that is Bayes consistent.
result Proves existing surrogates for disagreement discrepancy are not Bayes consistent.
New method discovers discrepancies between simplified models and experimental data.
problem Model discrepancies in nonlinear systems lead to significant deviations from true behavior.
method Sparse Identification of Nonlinear Dynamics (SINDy) algorithm to discover sparse model terms.
result Improvement in performance with a discrepancy model in simulations.
New method estimates model discrepancy without sampling for unnormalized models.
problem Evaluating and training unnormalized density models efficiently.
method Estimate Stein discrepancy using neural network parameterized vector function.
result Method outperforms existing goodness-of-fit tests and training methods.
Unsupervised domain adaptation is the problem setting where data generating distributions in the source and target domains are different, and labels in the target domain are unavailable. One important question in unsupervised domain adaptation is how to measure the difference between the source and target domains. A pr…