New model improves image captioning's ability to describe unseen concepts.
problem Image captioning models struggle with describing unseen combinations of concepts.
method Proposes a multi-task model combining caption generation and image-sentence ranking, with a decoding mechanism to re-rank captions based on image similarity.
result The model significantly outperforms state-of-the-art models in compositional generalization.
Survey of deep learning methods for image captioning.
problem Generating accurate and complex image descriptions.
method Comprehensive review of deep learning techniques for image captioning.
result Analysis of strengths, limitations, and popular datasets in deep learning image captioning.
Bayesian approach improves image captioning quality metrics.
problem Improving image captioning quality metrics.
method Bayesian Self-Critical Sequence Training (B-SCST) with Monte Carlo dropout approximate variational inference.
result B-SCST improves CIDEr-D scores on various image captioning datasets.
New 2D maps improve image captioning models.
problem Captions generated by RNNs are often poor.
method Used 2D maps instead of vectors to represent latent states.
result 2D maps lead to better captioning performance.
Generative model uses captions to generate images, improving semantic understanding.
problem Complex image generation models require large datasets and intricate learning.
method Adapts captioning models to generate images, using learned sentence and frame vectors.
result Images generated from multiple captions better capture semantic meaning.
C4Synth generates images from multiple captions to improve image quality.
problem Generating images from a single caption is limited; multiple captions are needed.
method Two deep generative models that ensure 'Cross-Caption Cycle Consistency'.
result Quantitative and qualitative validation on Caltech-UCSD Birds and Oxford-102 Flowers datasets.
Current image captioning systems miss spatial location details.
problem Capturing spatial location information in image captions.
method Evaluation of image captioning systems from literature.
result Language models alone are insufficient for capturing spatial location in image captions.
Improved image captions through adversarial semantic alignment.
problem Automatic evaluation and generalization to unseen compositions.
method Context-aware LSTM captioner and co-attentive discriminator for semantic alignment; SCST training method.
result SCST training method shows better performance in semantic score and human evaluation.
Model learns image-word associations from captions using contrastive learning.
problem Phrase grounding, associating image regions to caption words.
method Optimizing word-region attention to maximize mutual information, using language model guided word substitutions for negatives.
result Model achieves 76.7% accuracy on Flickr30K Entities benchmark, a 5.7% gain from weak supervision.
Experiment shows that certain factors predict which New Yorker cartoon captions are funniest.
problem Determining which captions are funniest in the New Yorker cartoon contest.
method Comparison of 12 automatic methods for identifying funniest captions.
result Negative sentiment, human-centeredness, and lexical centrality are key predictors of funniest captions.
Transformer model estimates keywords for better audio captioning.
problem Indeterminacy in word selection for audio events/scenes.
method Transformer-based model with keyword estimation.
result Achieved state-of-the-art performance in AAC.
System tackles indeterminacies in automated audio captioning.
problem Word selection and sentence length indeterminacies in automated audio captioning.
method Solves caption generation and sub-indeterminacy problems through multi-task learning to estimate keywords and sentence length.
result Model achieved 20.7 SPIDEr score, significantly outperforming baseline.
Develops a variational autoencoder for image, label, and caption modeling.
problem Deep learning of images, labels, and captions.
method Uses a Deep Generative Deconvolutional Network (DGDN) and a Convolutional Neural Network (CNN) for image and latent feature encoding.
result Able to predict labels or captions for new images using latent code distributions.
Improves text-to-image translation by using GANs and captioning networks.
problem Generating images that accurately reflect the meaning of a sentence.
method Uses cycle consistent adversarial networks and captioning networks to improve image generation.
result Significantly improved image quality compared to existing methods.
JECL clusters images and captions by jointly learning representations and assignments.
problem Clustering image-caption pairs with limited structured training data.
method Parallel encoders trained with clustering and alignment objectives, minimizing KL divergence and maximizing Jensen-Shannon divergence, with regularizers.
result JECL outperforms single-view and multi-view methods on large image-caption datasets.
Paper proposes method to generate images from text using GANs trained on uncaptioned images.
problem Limited captioned image datasets for text-to-image synthesis.
method Conditional GANs trained on uncaptioned images with an Image Captioning Module.
result Promising preliminary results compared to unconditional GANs.
Efficient video captioning model captures cross-modal interactions.
problem Capturing frame-level cross-modal interactions in video captioning.
method Proposes High-Order Cross-Modal Attention (HOCA) and Low-Rank HOCA.
result Low-Rank HOCA achieves state-of-the-art performance.
Paper develops a framework for generating coherent image captions using visual features and hierarchical topics.
problem Generating semantically coherent paragraphs to describe image content.
method Plug-and-play hierarchical-topic-guided image paragraph generation framework integrating visual extractor and deep topic model.
result Proposed models can distill interpretable multi-layer semantic topics and generate diverse and coherent captions.
Study improves image-caption retrieval by quantifying feature and posterior uncertainty.
problem Improving reliability in image-caption retrieval tasks with deep learning models.
method Quantified feature and posterior uncertainty for model averaging and reliability measure in image-caption retrieval.
result Consistent improvement in retrieval performance with different datasets and architectures.
System automatically searches for scientific features in Mars rover images.
problem Manual task planning for Mars rover images is time-consuming and error-prone.
method Deep image captioning network and automated task prioritization.
result System can prioritize images for transmission based on search tasks.
Study finds CLIP's caption-based learning outperforms image-only methods under certain conditions.
problem Comparing CLIP's performance with traditional image-only methods in learning transferable representations.
method Controlled comparison of CLIP and image-only methods using a dataset with descriptive captions and specific criteria.
result CLIP's caption-based learning outperforms image-only methods when certain conditions are met, but can be detrimental in others.
This work investigates upper bounds for visual captioning without extra effort.
problem Establishing performance upper bounds for visual captioning datasets.
method Decomposing visual captioning into two steps and constructing upper bounds.
result Current state-of-the-art models fall short of learned upper bounds.
Improved satellite image captions enhance descriptiveness without large models.
problem Extracting meaningful text from satellite imagery.
method Evaluated seven models on a large benchmark, extended vocabulary, and introduced a novel confusion matrix.
result Reduced model size by 100x without sacrificing accuracy, offering new deployment opportunities.
RL for image captions improved with a language prior.
problem Learning biases and large sample space issues in RL image captioning.
method Added a language prior to constrain the action space.
result RL with the language prior module performs better in readability and speed.
Enhances image captioning with novel context combination methods.
problem Improving machine learning for image captioning with structured learning and meaningful interpretation.
method Combines Feature Distribution Composition (FDC), Multiple Role Representation Crossover (MRRC) attention layers, and language decoder.
result Significantly improved image captioning performance (35.3%) and established new standards.
This paper improves image captioning by aligning visual attention with sentence generation.
problem Generating accurate and meaningful image captions from visual scenes.
method Region-based attention and scene factorization to align visual perception with sentence generation.
result Combining region-based attention and scene-specific contexts improves image captioning performance.
Improved video and movie description using multitask learning.
problem Lack of training data and poor generalization in video captioning.
method Multitask learning encoder-decoder framework for video sequences.
result Improved performance on multi-caption and single-caption datasets.
Generates realistic faces from detailed textual descriptions.
problem Face generation from fine-grained textual descriptions.
method Conditional GAN model with DC-GAN and GAN-CLS loss, using CelebA dataset with generated captions.
result Promising results in generating diverse face images from fine-grained textual descriptions.
Non-autoregressive model speeds up sequence generation tasks.
problem Efficiency in sequence generation tasks.
method Iterative refinement based on latent variable models and denoising autoencoders.
result Significant speedup in decoding with comparable quality.
Generative Score Inference improves uncertainty quantification for multimodal data.
problem Accurate uncertainty quantification in multimodal learning tasks.
method Generative Score Inference (GSI) uses synthetic samples to approximate conditional score distributions.
result GSI achieves state-of-the-art performance in hallucination detection and image captioning uncertainty estimation.
Compact RNNs reduce parameters and improve efficiency.
problem High computational cost of RNNs with large inputs.
method Block-Term Tensor Decomposition (BT-TD) to reduce RNN parameters.
result BT-RNN achieves better accuracy and faster convergence than standard RNNs.
Fine-tunes language models with captions for differential equation solving.
problem Overreliance on function data in operator learning.
method Integrates human knowledge through captions and fine-tunes language models.
result Significantly enhanced performance and reduced function data requirements.
Improves sequence generation by training a backward network.
problem Generating long-term dependencies in sequence models.
method Train a backward recurrent network to predict states of a forward model.
result Achieves 9% relative improvement in speech recognition and significant improvement in caption generation.
Paper tackles model failures producing outputs of undesirable length.
problem Model failures producing outputs of undesirable length.
method Develops a differentiable proxy objective and a verification approach.
result Can produce outputs 50 times longer than input and prove output length below a certain size.
Batch-normalized RHN improves gradient control in recurrent networks.
problem Gradient vanishing or exploding in recurrent networks.
method Batch normalization applied at each recurrence loop in RHN.
result Batch-normalized RHN converges faster and performs better.
Improves text-to-image generation with bidirectional capabilities.
problem Generating realistic images from text descriptions.
method Integrates text and image modalities using MMVR architecture with n-gram cost function and multiple sentences.
result Significant improvement in image quality over existing methods (over 20%).
CMT efficiently manages memory by inserting and querying memories in logarithmic time.
problem Managing large memory stores efficiently for quick access and updates.
method Designing a Contextual Memory Tree (CMT) that inserts and retrieves memories in logarithmic time.
result CMT improves classification algorithms and image-captioning tasks, demonstrating better computational efficiency.
Grad-CAM visualizes CNN predictions to explain model decisions.
problem Making CNN models transparent and understandable.
method Using class-specific gradient information to localize important regions.
result Improved understanding of CNN-based models, including image captioning and VQA.
Pretrained model improves visual dialog performance.
problem Improving performance in visual dialog tasks.
method Pretrained ViLBERT model on vision-language datasets, fine-tuned on VisDial.
result Best model outperforms prior work by more than 1% on NDCG and MRR.
Area attention allows dynamic area-based attention in memory.
problem Fixed attention granularity limits model performance.
method Area attention dynamically determines area shape and size via learning.
result Area attention improves performance on neural machine translation and image captioning.
Seq-CVAE learns a latent space for each word position to capture sentence intention.
problem Capturing diversity in image captioning models.
method Seq-CVAE learns a sequential latent space for each word position, mimicking future sentence summaries.
result Significantly improves diversity metrics on MSCOCO dataset compared to baselines.
Bayesian RNNs improve language and image tasks with better uncertainty estimates.
problem Improving uncertainty estimates in RNNs for better model performance.
method Variational Bayes scheme with truncated backpropagation and local gradient information.
result Bayesian RNNs outperform traditional RNNs on language and image tasks.
DMAE learns shared latent space from unpaired data.
problem Learning shared latent space from unpaired multi-modal data.
method Formulates cross-domain representation learning and object matching problem, optimizes autoencoders and pairing.
result Promising results in image captioning and unsupervised classifier learning.
Unified framework for generating meteorological time series from text.
problem Lack of large-scale, physically grounded multimodal datasets and architectures ignoring spectral-temporal structure.
method Introduce MeteoCap-3B dataset and MTransformer model.
result State-of-the-art generation quality, accurate cross-modal alignment, strong semantic controllability.
The paper proposes a method to learn from neighbor annotations to improve predictions in ambiguous tasks.
problem Ambiguity in structured prediction problems with large output spaces makes exhaustive annotation impractical.
method Proposes an objective to transfer supervision from neighboring examples to improve predictions.
result Consistent improvements over standard maximum likelihood training and baselines in multi-label classification and image-grounded sequence modeling tasks.
A new technique called fraternal dropout improves RNN performance.
problem Optimizing recurrent neural networks (RNNs) is harder than feed-forward networks.
method Train two identical RNNs with different dropout masks to encourage robust representations.
result Achieves state-of-the-art results on sequence modeling tasks and improves image captioning and semi-supervised learning.
Unified architecture for multi-modal multi-task learning using transformer.
problem Training multiple tasks concurrently with varying modalities.
method Spatio-temporal cache mechanism for multi-modal learning.
result Training multiple tasks together reduces model size by about three times.
This work uses image generation models to find vision model bugs.
problem Automatically discovering failures in vision models.
method Conditional text-to-image generation and captioning models.
result Demonstrated utility of large-scale generative models to find vision model bugs.