Gradient-based framework for optimizing text prompts in diffusion models.
problem Efficiently optimizing prompts in text-to-image diffusion models with large domain space and non-differentiable embeddings.
method Formulated as discrete optimization over language space, designed compact subspaces, and introduced shortcut text gradient.
result Empirically discovered prompts that enhance or destroy image faithfulness.
ZegOT uses optimal transport to zero-shot segment images with text prompts.
problem Zero-shot semantic segmentation with limited image-text alignment knowledge.
method ZegOT uses optimal transport to match multiple text prompts with frozen image embeddings.
result ZegOT achieves state-of-the-art performance in zero-shot semantic segmentation.
Automates zero-shot classification by scoring and weighting prompts.
problem Improving zero-shot accuracy through prompt ensembling.
method Automatic prompt scoring and weighting method.
result Method outperforms existing techniques on various benchmarks.
Improved text generation with discourse-aware soft prompts.
problem Efficient fine-tuning methods don't generalize across all generation tasks.
method Hierarchical blocking and attention sparsity on prefix parameters.
result Structured design of prefix parameters yields more coherent generations.
New method improves image generation for inverse problems using text prompts.
problem Suboptimal performance of existing latent diffusion models for inverse problems.
method Prompt tuning and latent variable projection to optimize text embeddings.
result P2L method outperforms existing methods on various inverse problem tasks.
OTSeg uses multi-prompt Sinkhorn attention to improve zero-shot semantic segmentation.
problem Leveraging pre-trained CLIP knowledge to align text embeddings with pixel embeddings.
method OTSeg employs Multi-Prompts Sinkhorn (MPS) and Multi-Prompts Sinkhorn Attention (MPSA) to enhance semantic feature matching.
result OTSeg achieves state-of-the-art performance in zero-shot semantic segmentation tasks.
Shorter adversarial prompts help protect LLMs from jailbreak attacks.
problem Protecting large language models from jailbreak attacks with long adversarial suffixes.
method Adversarial training on shorter adversarial suffixes to defend against longer adversarial suffixes.
result Aligning LLMs on shorter adversarial suffixes can effectively defend against jailbreak attacks with longer suffixes.
Paper proposes a faster, higher-quality RL method for text-to-image models.
problem Improving image quality and prompt alignment in RL post-training of text-to-image models.
method Online RL variant that reduces variance by sampling paired trajectories and optimizing flow velocity.
result Method converges faster and yields higher output quality and prompt alignment.
Deep learning is still not a very common tool in speaker verification field. We study deep convolutional neural network performance in the text-prompted speaker verification task. The prompted passphrase is segmented into word states - i.e. digits -to test each digit utterance separately. We train a single high-level f…
HawkesLLM models text generation with temporal influence, improving semantic alignment under limited memory.
problem Path-dependent uncertainty in agentic text-simulation systems.
method HawkesLLM framework separates temporal influence modeling from text generation, using a multivariate Hawkes process and a language model.
result HawkesLLM improves late-stage semantic alignment under a compact prompt-memory budget.
Adaptive testing segments watermarked text from LLMs.
problem Distinguishing LLM-generated text from human-written content.
method Generalized likelihood-based detection method adapted to inverse transform sampling, removing prompt estimation sensitivity.
result Effective and robust method for segmenting watermarked text.
Method detects if text is generated by a language model with watermarks.
problem Detecting if text is generated by a language model with watermarks.
method Randomization tests and change point detection techniques.
result Method ensures Type I and Type II error control and accurately identifies watermarked sub-strings.
Machine learning methods have recently achieved high-performance in biomedical text analysis. However, a major bottleneck in the widespread application of these methods is obtaining the required large amounts of annotated training data, which is resource intensive and time consuming. Recent progress in self-supervised …
New method predicts nonfactuality in LLM responses using semantic isotropy.
problem Assessing trustworthiness of long-form LLM responses efficiently.
method Semantic isotropy of text embeddings on the unit sphere.
result Higher semantic isotropy correlates with lower factual consistency.
ASK-NN detects distribution drifts in LLM-generated text.
problem Hallucinations and artificial text in LLM-generated outputs.
method Asymmetric two-sample test based on directed k-nearest-neighbor graph.
result ASK-NN is competitive with baselines on various benchmarks.
Method estimates LLM error rates using Pareto optimization.
problem Quantifying error rates in text-generating models.
method Pareto optimization for generating risk scores.
result Risk scores correlate well with true error rates.
Improved text-to-image alignment using iterative VQA feedback.
problem Misalignment between text prompts and generated images, especially for complex inputs.
method Decompose complex prompts into assertions, evaluate each using VQA, combine scores iteratively.
result Significantly higher correlation with human ratings compared to CLIP, BLIP scores.
PAC-Bayesian theory improves text-to-image models by enforcing alignment and generalization.
problem Text-to-image models struggle with complex prompts, misaligning modifiers and neglecting certain elements.
method Proposes a Bayesian approach with custom priors over attention distributions to enforce desirable properties.
result Achieves state-of-the-art results across multiple metrics on standard benchmarks.
Bayesian approach quantifies uncertainty in LLM evaluations.
problem Statistical uncertainty in evaluating LLM behavior.
method Bayesian evaluation of LLM behavior using probabilistic text generation strategies.
result Bayesian approach provides useful uncertainty quantification about LLM behavior.
Paper introduces SDM for detecting LLM hallucinations, improving on entropy tests.
problem Challenges of Large Language Models (LLMs) with non-factual, nonsensical responses.
method Joint clustering on sentence embeddings to measure semantic divergence between prompts and responses.
result SDM framework detects deeper form of arbitrariness in LLM responses.
Improved image synthesis with user scribbles and text prompts.
problem Inadequate details in generated images due to domain shift.
method Optimization problem formulation and cross-attention for control.
result Significant improvement in user satisfaction (85.32% higher).
Bayesian approach quantifies uncertainty in LLM-based systems.
problem Uncertainty quantification in LLM-based systems, especially for high-stakes applications.
method Interpreting prompts as parameters in a Bayesian model, using MHLP for inference.
result Improvements in predictive accuracy and uncertainty quantification on various benchmarks.
Study quantifies how LLMs capture higher-order statistical structure using cumulant expansion.
problem Understanding how LLMs internalize statistical structure during next-token prediction.
method Cumulant-expansion framework treating softmax entropy as perturbation around center distribution.
result Cumulants reveal distinct signatures for mathematical vs. general text prompts, quantifying feature-learning dynamics.
StealthRank subtly boosts LLM rankings without detectable anomalies.
problem Adversarial manipulation of LLM-driven ranking systems.
method Energy-based optimization with Langevin dynamics for stealthy prompt generation.
result StealthRank outperforms existing methods in covertly boosting rankings.
Study uses LLMs to simplify financial regulation interpretation.
problem Complex financial regulations are hard to interpret and implement.
method Developed prompts to guide LLMs in extracting key information from regulations.
result GPT-4 outperforms other LLMs in processing and executing regulatory requirements.
LIDS assesses LLM summaries with interpretable key words.
problem Challenges in evaluating the quality of LLM summaries.
method BERT-SVD-based direction metric and SOFARI for key word extraction.
result LIDS provides interpretable key words for layered themes.
This paper analyzes financial sentiment using LLMs and FinBERT, improving accuracy with few-shot examples.
problem Financial sentiment analysis for market evaluation.
method Application of large language models and FinBERT, with focus on prompt engineering and few-shot learning.
result GPT-4o achieves similar sentiment classification accuracy to FinBERT with fewer examples.
NegToMe uses images to guide text-based models away from unwanted visual elements.
problem Insufficient text-based adversarial guidance for complex visual concepts.
method Negative token merging (NegToMe) using visual features from reference images.
result Significantly enhances output diversity and reduces visual similarity to copyrighted content.
New framework improves interpretability of trainable prompts.
problem Improving task-specific LLM performance with soft prompts remains a black-box method.
method Developed a theoretical framework for evaluating interpretability of trainable prompts, inspired new objective functions.
result Found a fundamental trade-off between interpretability and task performance in trainable prompts.
AudioPaLM combines text and speech models to improve speech processing and translation.
problem Improving speech understanding and generation models.
method Unified multimodal architecture combining text and speech models.
result AudioPaLM significantly outperforms existing speech translation systems.
ChatGPT improves financial reasoning, overcoming biases in gold investment.
problem Improving financial reasoning and overcoming biases in investment decisions.
method Applied advanced prompt engineering and semantic news information to enhance LLMs' performance.
result ChatGPT with CoT prompt provides more explainable predictions and higher investment returns.
VERA-V uses variational inference to discover vulnerabilities in multimodal vision-language models.
problem Existing methods for jailbreaking vision-language models are brittle, limited, and focus on single attacks.
method VERA-V recasts jailbreak discovery as learning a joint posterior distribution over text-image prompts, using variational inference and three complementary strategies.
result VERA-V consistently outperforms state-of-the-art baselines, achieving up to 53.75% higher attack success rate.
LLMs struggle with zero-shot annotation tasks due to model-internalized priors.
problem Impact of model-internalized priors on LLM performance in zero-shot annotation tasks.
method Investigated three dimensions: familiarity, decision stickiness, and susceptibility to misaligned task definitions.
result Nearly two-thirds of zero-shot errors are resistant to correction, with a rescue rate of 34.8%. Definition-Specific Familiarity (DSF) shows a positive association with model performance.
Study on how attention in prompt-tuning affects large language models.
problem Limited theoretical understanding of prompt-tuning and attention in LLMs.
method Exploration of prompt-tuning for one-layer attention architectures, contextual mixture-models, and self-contained prompt-attention model.
result Softmax-prompt-attention is more expressive than self-attention and linear-prompt-attention under contextual data model.
Unified framework for generating meteorological time series from text.
problem Lack of large-scale, physically grounded multimodal datasets and architectures ignoring spectral-temporal structure.
method Introduce MeteoCap-3B dataset and MTransformer model.
result State-of-the-art generation quality, accurate cross-modal alignment, strong semantic controllability.
InfoPrompt improves soft prompt tuning by maximizing mutual information, leading to better performance.
problem High sensitivity of prompt tuning to initial conditions and insufficient task-relevant information.
method Develops an information-theoretic framework to maximize mutual information between prompts and model parameters, using novel loss functions.
result InfoPrompt accelerates convergence and outperforms traditional methods.
Analysis of pretrained models' effectiveness in downstream tasks.
problem Understanding why pretrained models perform well in NLP tasks.
method Analyzed head and prompt tuning approaches using latent variable models.
result Prompt tuning provides stronger guarantees than head tuning.
PDO optimizes LLM prompts without labels, improving performance.
problem Optimizing prompts for LLMs without access to labeled data.
method Pairwise preference feedback, dueling bandits, Thompson Sampling, mutation.
result PDO identifies stronger prompts than label-free methods.
Understanding optimal prompts for binary sequence predictors is challenging.
problem Finding good prompts for binary sequence predictors is difficult.
method Viewing prompting as finding the best conditioning sequence on a near-optimal sequence predictor, using empirical and statistical analysis.
result Optimal prompts can be better understood given the pretraining distribution, which is not usually available.
Study evaluates if LLMs have company-specific biases in financial sentiment analysis.
problem Evaluating if large language models exhibit company-specific biases in financial sentiment analysis.
method Comparing sentiment scores with and without company names, constructing economic models, and empirical analysis.
result LLMs show company-specific biases in sentiment analysis, impacting investor behavior and stock prices.
CAPO optimizes LLM prompts more efficiently and cost-effectively.
problem Costly and inefficient automatic prompt optimization for LLMs.
method Integrates AutoML techniques for evolutionary optimization of instructions and few-shot examples.
result Significantly improves prompt optimization efficiency and accuracy.
This paper formalizes AI safety using hypothesis testing in GenAI.
problem Ensuring safety of generative AI tools that create realistic content.
method Formalization of computational safety through hypothesis testing and signal processing.
result Demonstrates how AI safety can be assessed quantitatively using mathematical frameworks.
New text-to-image diffusion models improve scene understanding for AI agents.
problem Fine-grained scene understanding for AI agents from text and images.
method Pre-trained text-to-image diffusion models optimized for generating images from text prompts.
result Policies learned with Stable Control Representations outperform state-of-the-art approaches on various control tasks.
Generative Adapter adapts LMs with a single forward pass, reducing inference overhead.
problem Efficient adaptation of large language models for new contexts.
method Generative Adapter directly maps new contexts to low-rank LM adapters via self-supervised learning.
result Significant reduction in inference overhead with no need for fine-tuning.
Unified framework for studying softmax attention under large prompts.
problem Challenges in theoretical analysis of softmax attention.
method Measure-based framework for finite and infinite prompts.
result Softmax attention converges to linear attention in the large-prompt regime.
Diffusion LLMs can efficiently generate harmful prompts for adversarial testing.
problem Generating harmful prompts for adversarial testing is resource-intensive and costly.
method Transformed adversarial prompt optimization into an efficient inference task using pretrained Diffusion LLMs.
result Only a few conditional samples are required to generate harmful prompts with high reward.
PromptEval estimates LLM performance across many prompts, improving reproducibility.
problem Limited prompt templates affect LLM benchmark reproducibility.
method Estimates performance distribution across many prompts using borrowed strength.
result PromptEval accurately estimates performance quantiles with practical budget.
Transformer models can approximate smooth functions with prompts, enhancing LLMs' dynamic capabilities.
problem Lack of theoretical framework for prompt engineering in transformer models.
method Formal framework demonstrating transformer models can approximate β-times differentiable functions with prompts. result Transformer models can approximate β-times differentiable functions with arbitrary precision using appropriately structured prompts.