An important goal of computer vision is to build systems that learn visual representations over time that can be applied to many tasks. In this paper, we investigate a vision-language embedding as a core representation and show that it leads to better cross-task transfer than standard multi-task learning. In particular…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “vision-language tasks”
Robust CLIP improves vision models' resistance to attacks.
problem Vulnerability of vision models to adversarial attacks.
method Unsupervised adversarial fine-tuning of CLIP.
result Robust CLIP prevents stealth attacks on vision models.
Model learns image-word associations from captions using contrastive learning.
problem Phrase grounding, associating image regions to caption words.
method Optimizing word-region attention to maximize mutual information, using language model guided word substitutions for negatives.
result Model achieves 76.7% accuracy on Flickr30K Entities benchmark, a 5.7% gain from weak supervision.
This research predicts stock market movements using Vision-Language models.
problem Predicting future stock market direction using historical data.
method Utilizing image and byte-based representations of stock data processed with Vision-Language models.
result The proposed approach significantly outperforms deep learning baselines.
Zero-shot understanding of accidents from surveillance videos using vision-language models
problem Accident understanding from surveillance videos
method Three-stage pipeline with vision-language similarity, metadata-driven multi-prompt reasoning, and entropy-gated pairwise adjudicator
result Substantial improvement in harmonic-mean score over baseline
Study identifies key parameters and input dimensions making LLMs and VLMs brittle.
problem Vulnerability of large language and vision-language models to perturbations.
method Proposed FI measure based on information geometry to quantify sensitivity.
result Small subset of high FI parameters significantly contribute to brittleness.
The paper tests semantic importance in opaque models using betting.
problem Precise statistical guarantees for semantic concepts in black-box models.
method Formalizes global and local statistical importance via conditional independence and SKIT.
result Shows effectiveness and flexibility of the framework on various models.
VERA-V uses variational inference to discover vulnerabilities in multimodal vision-language models.
problem Existing methods for jailbreaking vision-language models are brittle, limited, and focus on single attacks.
method VERA-V recasts jailbreak discovery as learning a joint posterior distribution over text-image prompts, using variational inference and three complementary strategies.
result VERA-V consistently outperforms state-of-the-art baselines, achieving up to 53.75% higher attack success rate.
Unified model learns concepts across domains like left and right.
problem Limited generalization of language concepts in inference-only models.
method Logic-Enhanced Foundation Model (LEFT) with a differentiable, domain-independent program executor.
result LEFT flexibly learns and reasons with concepts across 2D images, 3D scenes, human motions, and robotic manipulation.
This work improves interpretability in deep learning models by introducing a two-level concept discovery framework.
problem High complexity and lack of interpretability in deep learning models, especially for safety-critical tasks.
method Concept Bottleneck Models (CBMs) framework combining vision-language models and data-driven coarse-to-fine concept selection.
result The proposed framework outperforms recent CBM approaches and provides a principled interpretability.
The study examines how much data is needed for generative and vision-language models to make reliable predictions.
problem Ensuring reliable predictions with low data for models used in medical decision support.
method Analyzes uniform convergence bounds for VLM-induced classifiers under low-dimensional semantic representations.
result Finite-sample uniform convergence bounds for accuracy and calibration functionals of VLM-induced classifiers.
Lipschitz RNNs improve stability and performance in various tasks.
problem Improving stability and performance of RNNs.
method Introduced a Lipschitz recurrent unit with a linear and Lipschitz nonlinear component for stability analysis.
result Lipschitz RNNs outperform existing units on benchmark tasks.
We present Vision-based Navigation with Language-based Assistance (VNLA), a grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments. The task emulates a real-world scenario in that (a) the requester may not know how to navigate to …
WebGUM learns web navigation from multimodal data, outperforming previous methods.
problem Limited generalization from domain-specific models in web navigation.
method Instruction-following multimodal agent trained on vision-language foundation models.
result Significant improvement in web navigation performance on benchmarks.
Unified framework reduces NFEs for inverse problems.
problem High computational costs and degraded reconstruction quality in existing LDM-based inverse solvers.
method Consistency Regularised Gradient Flows for posterior sampling and prompt optimization.
result Significantly reduced computational cost with state-of-the-art performance.
VLM judges rank well but score poorly; task difficulty and annotation quality affect interval width.
problem VLMs as judges lack reliability indicators in multimodal evaluations.
method Conformal prediction using score-token log-probabilities.
result Evaluation uncertainty is task-dependent, affecting interval width and reliability.
Paper proposes a new framework for hypothesis testing in imaging.
problem Challenges in hypothesis testing for imaging data.
method Combines self-supervised imaging, vision-language models, and non-parametric hypothesis testing.
result Demonstrates improved power and robust error control in image-based phenotyping.
Paper develops a theory explaining contrastive pre-training for multimodal AI.
problem Limited theoretical understanding of contrastive pre-training for multi-modal AI.
method Introduces approximate sufficient statistics and Joint Generative Hierarchical Model.
result Near-minimizers of contrastive loss are approximately sufficient, enabling diverse downstream tasks.
This paper tackles negative transfer in multi-task learning by introducing class-wise weights.
problem Negative transfer hampers function from achieving optimality in multi-task learning.
method Introduces class-wise weights to drive positive transfer and suppress negative transfer.
result Demonstrates improved performance in multi-task learning by reducing negative transfer.
Debiased contrastive learning improves representation learning by correcting for same-label sampling.
problem Sampling negative examples from truly different labels improves performance in self-supervised representation learning.
method Developed a debiased contrastive objective that corrects for the sampling of same-label datapoints without true labels.
result The proposed debiased contrastive objective consistently outperforms state-of-the-art methods across vision, language, and reinforcement learning benchmarks.
Dp-CLIP preserves privacy in multimodal AI training.
problem Privacy concerns in multimodal AI, especially in vision-language tasks.
method Differentially private adaptation of CLIP model.
result Dp-CLIP retains accuracy while ensuring privacy.
TRAK traces model predictions to training data efficiently.
problem Inefficiency in data attribution methods for large-scale models.
method TRAK: a new data attribution method that is both effective and computationally tractable.
result TRAK matches the performance of methods requiring thousands of models with just a handful.
Cosmos models scenes using neural encodings and symbolic attributes for compositional generalization.
problem Modeling scenes with high performance on unseen input scenes composed of known visual elements.
method Neurosymbolic grounding with neurosymbolic scene encodings and attention mechanisms.
result Establishes a new state-of-the-art for compositional generalization in world modeling.
Effective theory for Transformer initialization improves model performance.
problem Improving performance of Transformers at initialization.
method Effective-theory analysis of signal propagation in wide and deep Transformers.
result Particular width scalings of initialization and training hyperparameters.
CLIP learns joint image-text representations for zero-shot learning.
problem Understanding and improving zero-shot transfer performance in CLIP.
method Formal study of transferrable representation learning and analysis of zero-shot transfer performance.
result Proposes a new CLIP-type approach that outperforms existing methods.
AutoDiff combines auto-encoder and diffusion model for realistic tabular data synthesis.
problem Generating realistic synthetic tabular data with heterogeneous features.
method Employing auto-encoder architecture to handle tabular data's complexity.
result Synthetic tables from AutoDiff show good statistical fidelity and perform well in machine learning tasks.
CLIP models robustness to spurious features is re-evaluated using a new dataset.
problem Existing robustness tests of CLIP models may not fully reflect their performance on spurious features.
method Crafted a new dataset (CounterAnimal) to reveal CLIP models' reliance on realistic spurious features.
result CLIP models are robust to spurious features learned from their training data, not ImageNet.
PyFi uses adversarial agents to train VLMs on financial image understanding.
problem Training VLMs to understand complex financial questions.
method PyFi-600K dataset and adversarial MCTS mechanism.
result Fine-tuned VLMs improve by 19.52% and 8.06% on financial question accuracy.
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
Conformal Alignment ensures trustworthy outputs from foundation models.
problem Ensuring outputs from foundation models align with human values in high-stakes tasks.
method A framework that trains an alignment predictor using reference data to select trustworthy outputs.
result Conformal Alignment accurately identifies trustworthy outputs via lightweight training over moderate reference data.
New text-to-image diffusion models improve scene understanding for AI agents.
problem Fine-grained scene understanding for AI agents from text and images.
method Pre-trained text-to-image diffusion models optimized for generating images from text prompts.
result Policies learned with Stable Control Representations outperform state-of-the-art approaches on various control tasks.
Transformers can approximate Kalman Filtering in linear systems with small error.
problem Approximating Kalman Filtering using Transformers for linear dynamical systems.
method Two-step reduction: 1) Softmax self-attention block approximates Nadaraya-Watson kernel smoothing, 2) This estimator approximates Kalman Filter.
result Constructs a Transformer that implements the Kalman Filter with small additive error, uniformly bounded in time.
SC-InfoNCE improves InfoNCE for feature clustering in contrastive learning.
problem Lack of theoretical understanding of InfoNCE's feature clustering mechanism.
method Introduced a transition probability matrix to model data augmentation dynamics and optimize feature similarity.
result SC-InfoNCE achieves strong performance across diverse domains, aligning feature similarity with downstream data.
Paper extends 2D ZSAD to 3D MRI without training, achieving robust anomaly detection.
problem Challenges in extending zero-shot anomaly detection to 3D medical images.
method Constructs localized volumetric tokens by aggregating 2D slices processed by 2D foundation models.
result Training-free, batch-based ZSAD effectively extends from 2D encoders to full 3D MRI volumes.
MM-DREX adapts LLM experts for financial trading via dynamic routing.
problem Challenges of non-stationary financial markets and static expert designs.
method MM-DREX uses a VLM-powered dynamic router to allocate expert weights and designs heterogeneous trading experts.
result Significantly outperforms 15 baselines across key metrics.
Martingale Doppelgänger-Eval benchmarks VLMs on candlestick evidence vs. trend extrapolation
problem Auditing whether VLMs use chart evidence or trend extrapolation
method Proving formal limitations and designing controlled mechanisms
result Identifying regression coefficients for evidence vs. trend
A new method for differentiable structured sparsity improves neural network performance and sparsity.
problem Non-differentiability of structured sparsity penalties in neural networks.
method Introducing -Gating, a differentiable approach to structured overparameterization.
result The -Gating objective converges to the -regularized loss and induces sparse learning dynamics.
S2D selectively decays large singular values to improve quantization of neural activations.
problem Large activation outliers in transformer models cause accuracy drops during quantization.
method Selective Spectral Decay () that surgically regularizes only the largest singular values.
result Significantly reduces activation outliers and produces well-conditioned representations.
Optimal selective classification using likelihood ratios improves model reliability.
problem Enhancing predictive model reliability by allowing uncertain predictions.
method Neyman--Pearson lemma applied to likelihood ratios for optimal selection.
result Neyman--Pearson-informed methods outperform existing baselines under covariate shifts.
FADE adapts machine learning models to evolving data efficiently.
problem Sequential covariate shift in dynamic environments.
method FADE uses Fisher information geometry for robust learning under SCS.
result FADE achieves up to 19% higher accuracy under severe shifts.
DoRA improves adaptation efficiency for large models by factoring norms and fusing kernels.
problem High-rank DoRA is computationally expensive and infeasible on common GPUs.
method Factored norms and fused Triton kernels to reduce memory and speed up computation.
result Fused implementation is up to 2.0x faster for inference and 1.9x faster for gradient computation.
A new framework quantifies how model explanations influence each other.
problem Understanding how different model explanations interact and influence each other.
method Introducing the metagame, a conceptual framework for measuring second-order interaction effects of model explanations using Shapley values.
result Meta-attributions provide directional insights into how feature interactions influence model explanations.
Improved Sample Complexity For Diffusion Model Training Without Empirical Risk Minimizer Accesscs.LG
Improved sample complexity for diffusion models without needing empirical risk minimizers.
problem Theoretical limitations in sample complexity for diffusion models.
method Structured decomposition of score estimation error, eliminating dependence on neural network parameters.
result Achieved sample complexity bound of O(ε^(-4)) without empirical risk minimizer access.
Prior work in visual dialog has focused on training deep neural models on VisDial in isolation. Instead, we present an approach to leverage pretraining on related vision-language datasets before transferring to visual dialog. We adapt the recently proposed ViLBERT (Lu et al., 2019) model for multi-turn visually-grounde…
Diverse and accurate vision+language modeling is an important goal to retain creative freedom and maintain user engagement. However, adequately capturing the intricacies of diversity in language models is challenging. Recent works commonly resort to latent variable models augmented with more or less supervision from ob…
Paper proposes a faster, higher-quality RL method for text-to-image models.
problem Improving image quality and prompt alignment in RL post-training of text-to-image models.
method Online RL variant that reduces variance by sampling paired trajectories and optimizing flow velocity.
result Method converges faster and yields higher output quality and prompt alignment.
New framework for discrete-state diffusion models reduces sample complexity.
problem Lack of theoretical understanding and sample complexity analysis for discrete-state diffusion models.
method Developed a principled theoretical framework, decomposing score estimation error.
result Established sample complexity bound of .
This research analyzes how input and output layers affect deep neural networks' resistance to adversarial attacks.
problem The vulnerability of deep neural networks to adversarial inputs, especially non-gradient based attacks.
method Analysis of three different fully connected dense network classes with manipulated input and output layers.
result Manipulating input and output layers can significantly enhance a deep neural network's robustness against adversarial attacks.