A new robust prefix-tuning framework improves model robustness against adversarial attacks.
problem Lack of robustness in prefix-tuning for adversarial attacks.
method Leveraging layerwise activations of pretrained models for additional prefix finetuning during the test phase.
result Framework substantially improves robustness over strong baselines while maintaining comparable accuracy on clean texts.
CROP verifies clean prefixes in reasoning traces, improving downstream repair accuracy.
problem Uncertainty in reasoning traces prevents full certification of entire responses.
method CROP selects a calibrated threshold to certify the longest prefix with low risk proxies.
result CROP improves downstream repair accuracy by preserving valid reasoning and discarding misleading suffixes.
Prefix consistency improves model reliability by weighting answers based on their reproducibility.
problem Improving the reliability of large language models' reasoning traces.
method Use prefix consistency to weight candidate answers based on their reproducibility during regeneration.
result Prefix consistency is the best correctness predictor, reaching Standard MV plateau accuracy with up to 21x fewer tokens.
Improved text generation with discourse-aware soft prompts.
problem Efficient fine-tuning methods don't generalize across all generation tasks.
method Hierarchical blocking and attention sparsity on prefix parameters.
result Structured design of prefix parameters yields more coherent generations.
ReOPD uses pre-collected teacher trajectories to distill knowledge from multi-turn interactions.
problem The cost of fully online on-policy distillation for multi-turn interactions.
method ReOPD, an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes, addressing the prefix trap and distribution shift.
result ReOPD preserves or improves OPD-level accuracy, uses zero tool calls, and is at least 4imes faster per training step. Study finds exposure bias distortion is limited and not incremental in open-ended text generation.
problem Exposure bias in auto-regressive language models causing incremental distortion.
method Proposed metrics to quantify exposure bias impact, used ground-truth prefixes instead of model-generated prefixes.
result Exposure bias distortion is limited and not incremental during generation.
KVCOMM optimizes multi-agent LLM systems by reusing KV-caches, reducing redundant processing.
problem Substantial overhead from reprocessing overlapping contexts across multi-agent systems.
method KVCOMM reuses KV-caches and adjusts offsets for shared content using a pool of cached examples (anchors).
result Achieves over 70% reuse rate across diverse multi-agent tasks, up to 7.8x speedup.
Autostackability for finitely generated groups is defined via a topological property of the associated Cayley graph which can be encoded in a finite state automaton. Autostackable groups have solvable word problem and an effective inductive procedure for constructing van Kampen diagrams with respect to a canonical fini…
Study risk-constrained Kelly optimization for mutually exclusive outcomes, proving support invariance and developing a structured algorithm.
problem Risk-constrained Kelly optimization for mutually exclusive outcomes with explicit state prices.
method Analyzes the finite mutually exclusive outcome version of risk-constrained Kelly optimization with explicit state prices, proving support invariance and developing a structured algorithm.
result Support is invariant across CRRA parameter and drawdown-surrogate parameter in the overround regime.
This paper explains how to optimize prompts for model adaptation.
problem Understanding and optimizing prompt tuning for model adaptation.
method Bayesian view and meta-learning to explain prompt optimization.
result Optimal prompting can be studied formally as conditioning Bayesian predictors.
New algorithm reduces regret in private online learning with optimal gap-dependent rate.
problem Optimal gap-dependent regret rate for private stochastic decision-theoretic online learning.
method Horizon-free pure-DP algorithm with exponential block partitioning and softmax selection.
result Explicit regret bound of 1000⋅(ΔminlogK+εlogK). Improved bounds on acylindricity for right-angled Artin groups.
problem Bounding the acylindrical action of right-angled Artin groups on their extension graphs.
method Exploring lattice properties, studying prefixes of powers, and extending quasi-root uniqueness.
result Cardinality of r-quasi-stabilizer is bounded by a linear function of r. A new method improves language model extrapolation without changing training.
problem Improving language model performance on sequences not seen during training.
method Perturbation-based training procedure for language models.
result The method consistently improves out-of-support prediction.
We improve private training accuracy with learning rate schedules and matrix factorizations.
problem Private training with learning rate schedules and correlated noise.
method General upper and lower bounds for learning rate schedules, memory-efficient constructions, and schedule-aware factorizations.
result Schedule-aware factorizations improve accuracy in private training.
Researchers develop hyperbolic neural networks for improved data embedding.
problem Lack of hyperbolic neural network layers limits the use of hyperbolic embeddings in machine learning.
method Combining Möbius gyrovector spaces and Riemannian geometry of the Poincaré model to derive hyperbolic neural network layers.
result Hyperbolic sentence embeddings outperform or match Euclidean variants on textual entailment and noisy-prefix recognition tasks.
Graph neural network predicts natural paths in graphs.
problem Predicting natural paths in graphs.
method Graph Neural Network (Gretel) for path extrapolation.
result Gretel efficiently predicts and samples from future path distributions.
The paper identifies patterns in language model weights used for memorizing paragraphs.
problem Locating the specific mechanisms and weights used by language models to memorize paragraphs.
method Examined gradients and attention patterns in language models to identify memorized paragraphs.
result Gradients of memorized paragraphs have a distinguishable spatial pattern, and localized attention heads are involved in paragraph memorization.
Improved math problem solvers using Transformer networks and diverse notations.
problem Challenges in constructing accurate and automatic solvers for math word problems.
method Transformer networks trained to translate math word problems to arithmetic expressions in infix, prefix, and postfix notations. Pre-training on general text corpus to improve performance.
result Significant improvements in accuracy, up to 10% over previous state of the art.
The following discourse is inspired by the works on hyperbolic groups of Epstein, and Neumann/Reeves. Epstein showed that geometrically finite hyperbolic groups are biautomatic. Neumann/Reeves showed that virtually central extensions of word hyperbolic groups are biautomatic. We prove the following generalisation: Theo…
Stage-based hyper-parameter optimization reduces GPU-hours and training time.
problem Efficiently executing hyper-parameter optimization for deep learning models.
method Stage-based execution strategy to remove redundant computations.
result Stage-based execution outperforms trial-based method by up to 6.60 times in GPU-hours and 4.13 times in training time.
Improved sentence modeling using Suffix Bidirectional LSTM.
problem Sequential bias in BiLSTMs limits long-range dependencies.
method Encodes each suffix and prefix of a sequence in both forward and reverse directions.
result SuBiLSTM improves performance in various NLP tasks.
Improved financial VA intent classification accuracy.
problem Determining user intents for unseen open intents.
method Supervised pre-training of intent representations using prefix-tuning and fine-tuning.
result 1.63% - 2.07% higher accuracy on banking77 benchmark.
For the numerical solution of the American option valuation problem, we provide a script written in MATLAB implementing an explicit finite difference scheme. Our main contribute is the definition of a posteriori error estimator for the American options pricing which is based on Richardson's extrapolation theory. This e…
Braids can be represented geometrically as laminations of punctured disks. The geometric complexity of a braid is the minimal complexity of a lamination that represents it, and tight laminations are representatives of minimal complexity. These laminations give rise to a normal form of braids, via a relaxation algorithm…
PPT optimizes transformer behavior by steering its latent posterior using prior samples.
problem Eliciting desired behavior from transformers without backpropagation.
method Posterior Prefix Tuning (PPT) uses predictive Monte Carlo (PMC) samples and importance sampling to optimize the latent posterior.
result PPT optimizes transformer behavior without backpropagation, achieving high utility across different utility functions.
Power-SMC reduces inference latency for training-free LLM reasoning.
problem Training-free LLM reasoning with low latency.
method Power-SMC, a training-free Sequential Monte Carlo scheme targeting sequence-level power distribution.
result Power-SMC reduces inference latency from 16-28× to 1.4-3.3× over baseline decoding.
Training-free method improves large language model sequence quality via reward-guided sampling.
problem Optimizing large language model sequence quality over token likelihood.
method Reward-augmented target distribution combined with Sequential Monte Carlo sampling.
result Significant gains in sequence generation and mathematical reasoning tasks.
New method optimizes language model performance for test-time strategies.
problem Mismatch between training objectives and test-time deployment of large language models.
method Tail-Extrapolated estimators to approximate best-of-N performance from limited training rollouts.
result Improved performance of best-of-N deployment across various models and datasets.
The paper tackles machine unlearning by designing efficient algorithms for adaptive query classes.
problem Designing efficient unlearning algorithms for machine learning models.
method Formalizes the problem and gives efficient unlearning algorithms for linear and prefix-sum query classes.
result Improved guarantees for stochastic convex optimization with reduced unlearning query complexity.
The paper parallelizes HMM inference for efficient long-term computations.
problem Efficiently computing inference in long-term hidden Markov models.
method Parallelization using associative elements and operators for sum-product and max-product algorithms.
result The proposed parallel algorithms are computationally efficient for long time horizons.
Supersymmetry contains initially noninvertible objects, but it is common to deal with the invertible ones only, factorizing former in some extent. We propose to reconsider this ansatz and try to redefine such fundamental notions as supermanifolds, fiber bundles and homotopies using some weakening invertibility conditio…
The paper diagnoses factor models using characteristic axes and zero-curve restrictions.
problem Tackles systematic sign reversals and overcorrections in factor model pricing errors.
method Extends cap-axis integral diagnostic to general characteristic axes, measuring pricing errors as bridge-alpha curves.
result Axis-level pricing errors are nearly orthogonal to maximum-Sharpe gains, showing systematic sign reversals and overcorrections.
The paper diagnoses factor-model pricing errors using characteristic axes and bridge-alpha curves.
problem Tackles systematic sign reversals and overcorrections in factor-model pricing errors.
method Extends cap-axis integral diagnostic to characteristic axes, measures pricing errors as bridge-alpha curves, and uses a predetermined characteristic order to generate zero-curve restrictions.
result Axis-level pricing errors are nearly orthogonal to maximum-Sharpe gains, showing significant sign reversals and overcorrections.
Learnable token perturbations boost extrapolation in LLMs.
problem Limited flexibility of current discrete perturbations in large language models.
method Learnable continuous latent vector transformations in embedding space, unbiased estimating equations, stochastic gradient descent optimization.
result Significant gains in out-of-domain settings over state-of-the-art methods.
The ECME algorithm has proven to be an effective way of accelerating the EM algorithm for many problems. Recognising the limitation of using prefixed acceleration subspace in ECME, we propose the new Dynamic ECME (DECME) algorithm which allows the acceleration subspace to be chosen dynamically. Our investigation of an …
This paper diagnoses factor-model pricing errors using a new method.
problem Measuring pricing errors in factor models with general characteristic axes.
method Developed a method to measure factor-model pricing errors as bridge-alpha curves, using a predetermined characteristic order and prefix portfolios.
result Adding a counterpart factor flips the curve's sign on every axis, but only HML and CMA overcorrect enough to be rejected.
We study the rational permutation braids, that is the elements of an Artin-Tits group of spherical type which can be written x−1y where x and y are prefixes of the Garside element of the braid monoid. We give a geometric characterization of these braids in type An and Bn and then show that in spherical …
A greedy active learning algorithm for logistic regression reduces model size and training size.
problem Binary classification with reduced model size and training size.
method Modified batch subject selection strategy with greedy variable selection.
result Competitive performance with smaller training size and model size.
An estimate on the number of distinct relative periodic orbits around a stable relative equilibrium in a Hamiltonian system with continuous symmetry is given. This result constitutes a generalization to the Hamiltonian symmetric framework of a classical result by Weinstein and Moser on the existence of periodic orbits …
Predicts node sequences in graphs using multi-order network models.
problem Predicting sequences of node traversals in graphs.
method Combines multiple higher-order network models into a multi-order model, fitting and selecting the optimal maximum order.
result Outperforms state-of-the-art algorithms for next-element and full sequence prediction.
The paper tackles biases in session-based recommender systems by modeling user interest as a stochastic process.
problem Data uncertainty, popularity bias, and exposure bias in session-based recommender systems.
method The paper proposes treating user interest as a stochastic process in the latent space, debiasing item embeddings, modeling dense user interest, and introducing fake targets to simulate extended exposure.
result The proposed approach mitigates challenges in session-based recommender systems, as shown by computational experiments on various datasets.
We provide an algorithm to solve the word problem in all fundamental groups of closed 3-manifolds; in particular, we show that these groups are autostackable. This provides a common framework for a solution to the word problem in any closed 3-manifold group using finite state automata. We also introduce the notion of a…
Proving that next-token prediction makes language models generate coherent long documents.
problem Understanding why language models generate coherent documents despite focusing on next-token prediction.
method Proving the power of next-token prediction in learning longer-range structure using Recurrent Neural Networks (RNN).
result Optimizing next-token prediction in RNNs yields a model that closely approximates the training distribution, even for long-range coherence.
PathBoost boosts graph-level predictions using path-based features.
problem Graph-level classification and regression challenges.
method Gradient tree boosting method for graph-level prediction.
result PathBoost outperforms graph neural networks and graph kernel approaches in many cases.
LoBoost improves local conformal prediction for gradient-boosted trees without extra data splits.
problem Quantifying uncertainty in gradient-boosted tree predictions.
method Model-native local conformal prediction using leaf structure.
result Competitive interval quality and improved test MSE with large calibration speedups.
AutoGraph uses transformers to efficiently generate graphs as sequences.
problem Efficiently generating large, sparse graphs without expensive node features.
method Flattening graphs into sequences and using decoder-only transformers.
result AutoGraph achieves state-of-the-art performance on synthetic and molecular benchmarks.
The trimming scheme with a prefixed cutoff portion is known as a method of improving the robustness of statistical models such as multivariate Gaussian mixture models (MG- MMs) in small scale tests by alleviating the impacts of outliers. However, when this method is applied to real- world data, such as noisy speech pro…
Neural networks with learned biases can approximate any function.
problem Whether neural networks with only learned biases can approximate any continuous function.
method Theoretical and numerical analysis of random weights and learned biases in neural networks.
result Feedforward and recurrent neural networks with random weights can approximate any continuous function and dynamical systems.