For bidirectional joint image-text modeling, we develop variational hetero-encoder (VHE) randomized generative adversarial network (GAN), a versatile deep generative model that integrates a probabilistic text decoder, probabilistic image encoder, and GAN into a coherent end-to-end multi-modality learning framework. VHE…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “image-text models”
Unsupervised methods have proven effective for discriminative tasks in a single-modality scenario. In this paper, we present a multimodal framework for learning sparse representations that can capture semantic correlation between modalities. The framework can model relationships at a higher level by forcing the shared …
Proposes MR-SNE for multimodal data visualization.
problem Visualizing data from multiple domains with relations across them.
method Extends t-SNE to compute augmented relations and jointly embed them in a low-dimensional space.
result Demonstrates promising performance in visualizing Flickr and Animal with Attributes 2 datasets.
Proposes a framework for generating explainable AI exemplars.
problem Need to explain decisions of complex deep learning models.
method Generative model with evolutionary strategy to synthesize exemplars.
result Framework is generic and model-agnostic for various data types.
Estimates CATEs for structured treatments using a new decomposition method.
problem Estimating conditional average treatment effects for complex data types.
method Generalized Robinson decomposition, isolating causal estimand, arbitrary model plugging, quasi-oracle convergence guarantee.
result Demonstrates superior performance in CATE estimation compared to prior work.
Deep Learning has been shown to learn efficient representations for structured data such as image, text or audio. In this chapter, we present neural network architectures that are able to learn efficient representations of molecules and materials. In particular, the continuous-filter convolutional network SchNet accura…
PoGDiff improves diffusion models on imbalanced datasets.
problem Diffusion models struggle with imbalanced text-image pairs.
method PoGDiff replaces ground-truth with a Product of Gaussians.
result PoGDiff enhances generation accuracy and quality on imbalanced datasets.
While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations fro…
Traditional clustering methods often perform clustering with low-level indiscriminative representations and ignore relationships between patterns, resulting in slight achievements in the era of deep learning. To handle this problem, we develop Deep Discriminative Clustering (DDC) that models the clustering task by inve…
Surrogate explainers of black-box machine learning predictions are of paramount importance in the field of eXplainable Artificial Intelligence since they can be applied to any type of data (images, text and tabular), are model-agnostic and are post-hoc (i.e., can be retrofitted). The Local Interpretable Model-agnostic …
Adversarial attacks on deep neural networks traditionally rely on a constrained optimization paradigm, where an optimization procedure is used to obtain a single adversarial perturbation for a given input example. In this work we frame the problem as learning a distribution of adversarial perturbations, enabling us to …
Classifies and constructs intertwining differential operators between line and vector bundles over real projective space.
problem Classifying and constructing intertwining differential operators between line and vector bundles over real projective space.
method F-method for classification and construction of intertwining differential operators.
result Generalizes a classical result of Bol for and classifies intertwining operators for .
CLIP learns joint image-text representations for zero-shot learning.
problem Understanding and improving zero-shot transfer performance in CLIP.
method Formal study of transferrable representation learning and analysis of zero-shot transfer performance.
result Proposes a new CLIP-type approach that outperforms existing methods.
The paper classifies and constructs differential symmetry breaking operators from a line bundle to a vector bundle over real projective spaces.
problem Classifying and constructing differential symmetry breaking operators.
method Utilizing factorization identities and branching laws of generalized Verma modules.
result Differential symmetry breaking operators from a line bundle to a vector bundle over real projective spaces are classified and constructed.
Global pooling, such as max- or sum-pooling, is one of the key ingredients in deep neural networks used for processing images, texts, graphs and other types of structured data. Based on the recent DeepSets architecture proposed by Zaheer et al. (NIPS 2017), we introduce a Set Aggregation Network (SAN) as an alternative…
We propose JECL, a method for clustering image-caption pairs by training parallel encoders with regularized clustering and alignment objectives, simultaneously learning both representations and cluster assignments. These image-caption pairs arise frequently in high-value applications where structured training data is e…
This research explores how different discrete diffusion kernels affect graph generation quality.
problem The impact of different discrete diffusion kernels on graph generation quality.
method Developed a family of discrete diffusion kernels that converge to different Bernoulli priors.
result The quality of generated graphs is sensitive to the prior used, challenging previous intuitions.
Paper introduces OARF benchmark suite for federated learning systems.
problem Limited diversity in federated learning benchmarks.
method Characterizes OARF benchmark suite with diverse data and applications.
result Federated learning can effectively increase end-to-end throughput.
Successful multimodal search and retrieval requires the automatic understanding of semantic cross-modal relations, which, however, is still an open research problem. Previous work has suggested the metrics cross-modal mutual information and semantic correlation to model and predict cross-modal semantic relations of ima…
Captum library simplifies model interpretability for PyTorch.
problem Improving model interpretability for various PyTorch models.
method Unified, open-source library with generic attribution algorithms and metrics.
result Unified and scalable model interpretability for multimodal inputs.
The paper discusses quantifying realism in generated images.
problem Designing functions to reliably distinguish realistic data from unrealistic data.
method Drawing on insights from algorithmic information theory, the paper introduces the notion of a universal critic.
result A good generative model alone is insufficient to solve the problem of realism quantification.
Generates coherent storybooks from plain text using diffusion models.
problem Ensuring coherency in a sequence of images for storytelling applications.
method Combines pre-trained LLM and text-guided Latent Diffusion Model for zero-shot generation.
result Outperforms state-of-the-art image editing baselines in generating coherent storybooks.
Leveraging Local Variation in Data: Sampling and Weighting Schemes for Supervised Deep Learningstat.ML
New method improves neural network performance by focusing on steep function regions.
problem Improving neural network performance by focusing on steep function regions.
method Variance Based Samples Weighting (VBSW) using labels local variance to weight training points.
result Significantly increases the performances of neural networks for various tasks.
FHDMs achieve optimal convergence in spherically supported data.
problem Statistical convergence properties of FHDMs for spherical data.
method FHDMs leverage random generation time and Doob's h-transform to optimize convergence rate.
result Achieve minimax optimal convergence rate in total variation for spherically supported Sobolev smooth data.
DreamFusion uses text-to-image diffusion models to create 3D images efficiently.
problem Lack of large-scale 3D datasets and efficient architectures for 3D synthesis.
method Adapting a 2D diffusion model to 3D synthesis using a loss based on probability density distillation.
result A 3D model can be optimized from a 2D diffusion model, allowing for text-to-3D synthesis.
PyKale bridges interdisciplinary ML with Python, enabling accurate predictions.
problem Cross-disciplinary barriers in machine learning.
method Knowledge-aware machine learning on graphs, images, texts, and videos.
result Enables multimodal learning and transfer learning with latest deep learning models.
Transformer is a popularly used neural network architecture, especially for language understanding. We introduce an extended and unified architecture that can be used for tasks involving a variety of modalities like image, text, videos, etc. We propose a spatio-temporal cache mechanism that enables learning spatial dim…
BC-LLM uses LLMs to find concepts without predefined sets, improving interpretability and performance.
problem Finding a balance between interpretability and accuracy in concept extraction models.
method Bayesian approach with LLMs as both concept extractor and prior.
result BC-LLM outperforms interpretable and black-box models across various datasets.
Masking diffusion outperforms other discrete diffusion models by incorporating jump times into the model.
problem Improving the performance of discrete diffusion models.
method Conditioning on the jump schedule of discrete Markov processes.
result Schedule-conditioned discrete diffusion (SCUD) models outperform classical and masking diffusion models.
Adversarial Robustness Toolbox (ART) is a Python library supporting developers and researchers in defending Machine Learning models (Deep Neural Networks, Gradient Boosted Decision Trees, Support Vector Machines, Random Forests, Logistic Regression, Gaussian Processes, Decision Trees, Scikit-learn Pipelines, etc.) agai…
We study the problem of controllable generation of long-term sequential behaviors, where the goal is to calibrate to multiple behavior styles simultaneously. In contrast to the well-studied areas of controllable generation of images, text, and speech, there are two questions that pose significant challenges when genera…
In this paper we study output coding for multi-label prediction. For a multi-label output coding to be discriminative, it is important that codewords for different label vectors are significantly different from each other. In the meantime, unlike in traditional coding theory, codewords in output coding are to be predic…
The "Law" of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalitiescs.LG
The paper proves probabilistic alignment between unseen modalities using contrastive learning.
problem Aligning unseen modalities in unsupervised learning.
method Bayesian approach and direct comparison of contrastive representations.
result Direct comparison of contrastive representations recovers the same likelihood ratio as probabilistic graphical models.
Contrastive learning estimates transition kernels for continuous-time stochastic processes.
problem Estimating transition kernels for continuous-time stochastic processes without labeled data.
method Contrastive learning applied to strong-mixing continuous-time stochastic processes.
result Contrastive learning can estimate transition kernels for small-to-mid-range intervals in the diffusion case.
Deep learning improves survival analysis for complex data types.
problem Limited application of DL in survival analysis for complex data.
method Comprehensive review of DL methods for time-to-event analysis.
result Methods often ignore complex settings like multiple risks and censoring.
Study shows how to reduce data needed for learning under geometric constraints.
problem Learning high-dimensional data with geometric priors.
method Spherical harmonic decompositions and kernel methods for invariance and geometric stability.
result Improvements in sample complexity by leveraging group invariance, with asymptotic behavior depending on spectral properties.
We study how well one can recover sparse principal components of a data matrix using a sketch formed from a few of its elements. We show that for a wide class of optimization problems, if the sketch is close (in the spectral norm) to the original data matrix, then one can recover a near optimal solution to the optimiza…
FinZero improves financial time series forecasting accuracy with multimodal modeling.
problem Lack of interpretability, uncertainty, and scalability in financial time series forecasting.
method Developed a multimodal pre-trained model FinZero using UARPO method for reasoning, prediction, and uncertainty analysis.
result FinZero achieves an approximate 13.48% improvement in prediction accuracy over GPT-4o in high-confidence group.
SFM matches flows on statistical manifolds for better discrete generation.
problem Discrete generation on statistical manifolds with strong prior assumptions.
method Statistical Flow Matching (SFM) on manifold of categorical distributions using Fisher information metric.
result SFM achieves higher sampling quality and likelihood than other models.
SKADA-bench evaluates unsupervised DA methods across diverse modalities.
problem Evaluating unsupervised DA methods on diverse modalities with realistic validation.
method Nested cross-validation and unsupervised model selection scores.
result Highlights the importance of realistic validation and provides practical guidance.
Space Narrative: Generating Images and 3D Scenes of Chinese Garden from Text using Deep Learningcs.CV
Generates garden paintings from text descriptions using deep learning.
problem Lack of firsthand material for traditional Chinese garden reconstruction.
method Deep learning model trained on text and paintings of Ming Dynasty gardens.
result Model generates garden paintings in Ming Dynasty style based on textual descriptions.
Heterogeneous domain adaptation (HDA) aims to facilitate the learning task in a target domain by borrowing knowledge from a heterogeneous source domain. In this paper, we propose a Soft Transfer Network (STN), which jointly learns a domain-shared classifier and a domain-invariant subspace in an end-to-end manner, for a…
Deep generative models have been praised for their ability to learn smooth latent representation of images, text, and audio, which can then be used to generate new, plausible data. However, current generative models are unable to work with molecular graphs due to their unique characteristics-their underlying structure …
A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Modelscs.LG
Automates zero-shot classification by scoring and weighting prompts.
problem Improving zero-shot accuracy through prompt ensembling.
method Automatic prompt scoring and weighting method.
result Method outperforms existing techniques on various benchmarks.
ZegOT uses optimal transport to zero-shot segment images with text prompts.
problem Zero-shot semantic segmentation with limited image-text alignment knowledge.
method ZegOT uses optimal transport to match multiple text prompts with frozen image embeddings.
result ZegOT achieves state-of-the-art performance in zero-shot semantic segmentation.
Graphs represent natural and artificial systems; ML can learn from them.
problem Representing and analyzing complex systems using graphs.
method Graph Neural Networks (GNNs) for learning from graph data.
result GNNs enable learning from diverse graph-based data.
Datasets such as images, text, or movies are embedded in high-dimensional spaces. However, in important cases such as images of objects, the statistical structure in the data constrains samples to a manifold of dramatically lower dimensionality. Learning to identify and extract task-relevant variables from this embedde…
Adaptive weighting schemes enhance time-series data augmentation for financial and UCR datasets.
problem Limited size of time-series datasets hinders model performance.
method Two adaptive weighting schemes for automatic data augmentation.
result Improves annualized returns by over 50% on financial dataset and outperforms state-of-the-art on half of UCR datasets.