GluonCV and GluonNLP simplify deep learning for CV and NLP.
problem Lack of easy-to-use deep learning tools for CV and NLP.
method Developed modular APIs and pre-trained models for rapid prototyping.
result Facilitates rapid prototyping and reproducible research in CV and NLP.
GSA-Nets apply group equivariance to self-attention for vision tasks.
problem Improving self-attention networks for vision tasks.
method Define group-equivariant positional encodings.
result GSA-Nets outperform non-equivariant self-attention networks on vision benchmarks.
Sparse Vision MoE matches dense networks in image recognition while using less compute.
problem Scaling vision models efficiently in computer vision.
method Vision MoE (V-MoE) - a sparse version of Vision Transformer.
result V-MoE matches state-of-the-art dense networks in image recognition with half the compute.
Vision and language tasks often fail to test AI comprehensively.
problem Current vision and language tasks are flawed due to dataset and evaluation issues.
method Review of current state and proposal for improvement.
result State-of-the-art systems perform well due to dataset and evaluation flaws.
Computer vision models are unstable due to task symmetries and labelling issues.
problem Instability of computer vision models in classification tasks.
method Analysis of symmetries, categorical nature, and labelling issues.
result Instability is a necessary result of current computer vision formulation.
Mobile V-MoEs scale down ViTs for resource-constrained vision tasks.
problem Scaling down Vision Transformers for resource-constrained applications.
method Sparse Mixture-of-Experts (MoEs) applied to entire images, with a stable training procedure.
result Mobile V-MoEs achieve better performance-efficiency trade-offs than dense ViTs.
Describes explaining neurons in deep representations using compositional logical concepts.
problem Interpreting neuron behavior in deep neural networks.
method Identifying compositional logical concepts that closely approximate neuron behavior.
result Compositional explanations provide insights into model performance and allow for adversarial example creation.
Meta Omnium benchmarks few-shot learning across diverse vision tasks.
problem Evaluating generalization of few-shot learning across multiple vision tasks.
method Introduction of Meta Omnium dataset and evaluation of meta-learning algorithms.
result Meta-learning algorithms can generalize across diverse vision tasks.
Regularizes attention scores in vision transformers using bootstrapping.
problem Noisy and diffused attention maps in ViT limit interpretability.
method Statistical learning techniques, bootstrapping of attention scores.
result Improves shrinkage and sparsity of attention scores.
This work characterizes benign overfitting in Vision Transformers.
problem Understanding generalization of Vision Transformers when trained to overfit.
method Gradient descent on a data distribution model, focusing on self-attention layer and softmax.
result Established a condition to distinguish between small and large test errors based on signal-to-noise ratio.
Integrating visual and linguistic information into a single multimodal representation is an unsolved problem with wide-reaching applications to both natural language processing and computer vision. In this paper, we present a simple method to build multimodal representations by learning a language-to-vision mapping and…
Survey of self-supervised learning methods in computer vision, NLP, and graph learning.
problem Manual labeling and vulnerability to attacks in supervised learning.
method Generative, contrastive, and generative-contrastive approaches.
result Self-supervised learning achieves high performance in representation learning.
Generates detailed fashion feedback from outfit images.
problem Creating informative and diverse fashion feedback from outfit images.
method Trained deep generative models with visual attention, then improved with Maximum Mutual Information objective function.
result Generated sentences are more diverse and detailed.
Graph Structured Prediction Energy Networks model correlations for joint inference.
problem Joint inference over multiple variables with high-order correlations.
method Energy Networks for modeling explicit local and implicit higher-order correlations.
result Tractable inference with explicit modeling of correlations.
AI models aligned with human vision perform well on few data tasks.
problem Few-shot learning performance with limited data.
method Information-theoretic analysis and empirical testing of 491 models.
result Highly aligned models show better robustness to attacks and domain shifts.
Personalized education at scale improves student outcomes.
problem Traditional personalized education is expensive and inequitable.
method Adapting educational presentations using RL, semi-supervised learning, NLP, and CV.
result Personalized education at scale can improve student outcomes.
Paper uses 'manifold attack' to improve model accuracy and robustness.
problem Improving model performance with limited training data and large model parameters.
method Enforces manifold preservation from original data into latent space using adversarial learning.
result Improves accuracy rate and robustness to adversarial examples.
Zero-shot understanding of accidents from surveillance videos using vision-language models
problem Accident understanding from surveillance videos
method Three-stage pipeline with vision-language similarity, metadata-driven multi-prompt reasoning, and entropy-gated pairwise adjudicator
result Substantial improvement in harmonic-mean score over baseline
Paper introduces a multi-stage influence function to track model predictions.
problem Improving natural language processing and computer vision performance.
method Develops a multi-stage influence function to track predictions from finetuned models back to pretraining data.
result Identifies pretraining examples contributing most to finetuning task predictions.
This research integrates attention into XAI frameworks for better model explanations.
problem Improving the interpretability of transformer models.
method Developed two novel explanation methods: Shapley value decomposition and token-level directional derivatives.
result Attention weights can be meaningfully incorporated into XAI frameworks, enhancing transformer explainability.
Proposes deep collective learning to learn inputs and weights together in neural networks.
problem The need to optimize inputs alongside weights in deep neural networks.
method Introduces deep collective learning to learn inputs and weights jointly, using lookup tables.
result Demonstrates advantages and promise of Lookup-VNets and deep collective learning.
Deep transfer learning improves malware classification speed and accuracy.
problem Static malware classification accuracy and speed.
method Transfer learning from computer vision to static malware detection.
result Our method outperforms classical machine learning methods in accuracy, false positive rate, true positive rate, and F1 score.
Hierarchical Foresight improves robot vision tasks by planning long-term goals.
problem Compounding uncertainty and scalability issues in long horizon video prediction.
method Subgoal generation and planning using hierarchical visual foresight (HVF).
result Achieves nearly 200% performance improvement in vision-based manipulation tasks.
The thesis introduces methods to use semantic hierarchy in image classification.
problem Limited work in training image classifiers with non-conventional external guidance.
method Injects label hierarchy knowledge into arbitrary classifiers and uses order-preserving embeddings for image classification.
result Both embedding-based models and CNN-classifiers with hierarchical information outperform a hierarchy-agnostic model.
A computer vision approach improves neutral particle detection in particle flow algorithms.
problem Optimal reconstruction of particle content and kinematics in calorimeter images.
method Computer vision techniques applied to calorimeter images, using deep learning and super-resolution.
result Significantly improved reconstruction of neutral particle calorimeter energy deposits.
ResMem improves model generalization by explicitly memorizing residuals.
problem Improving model generalization in neural networks.
method ResMem algorithm that augments a model with a k-nearest neighbor based regressor fitted to residuals.
result ResMem consistently improves test set generalization across various benchmarks.
Survey of complex-valued neural networks for improved performance.
problem Lack of complex-valued neural networks in machine learning frameworks.
method Literature review of CVNNs.
result Advantages of CVNNs over real-valued neural networks.
NIPS 2018 Adversarial Vision Challenge aims to improve machine vision models.
problem Improving robustness of machine vision models and adversarial attacks.
method Organized a competition to measure progress in adversarial machine vision.
result Facilitated measurable progress in adversarial machine vision.
New datasets make AI models fail reliably, highlighting weaknesses.
problem AI models' vulnerabilities to adversarial examples.
method Created challenging datasets with adversarial filtering, tested on various models.
result Existing models perform poorly on new datasets, revealing shared weaknesses.
A novel method for visual question answering using scene graphs and reinforcement learning.
problem Answering free-form questions about images with deep linguistic and visual understanding.
method Context-driven, sequential reasoning based on scene graphs and reinforcement learning.
result Our method almost reaches human performance on the GQA dataset.
A benchmark for NLP models trained on text datasets.
problem Limited access to high-performance clusters for NAS experiments.
method Created a search space for recurrent neural networks on text datasets and trained 14k architectures.
result Demonstrated the potential of precomputed NAS results for NLP.
A new multi-hop FiLM approach improves visual reasoning tasks.
problem Challenging multi-modal tasks like visual question-answering and dialogue.
method Generate FiLM layer parameters in a multi-hop fashion, alternating between attending to language and generating parameters.
result Multi-hop FiLM generation achieves state-of-the-art performance on visual dialogue tasks.
This paper reviews and proposes a unified framework for contrastive learning.
problem The origins and development of contrastive learning across various fields.
method A comprehensive literature review and a general Contrastive Representation Learning framework.
result A unified framework simplifies and unifies contrastive learning methods.
A new system combines vision and language for person re-identification.
problem Real-world surveillance lacks visual data for person re-identification.
method Two-stream CNN framework with shared logits, CCA for modalities, multi-modal testing protocol.
result 22% improvement in re-identification performance with multi-modal queries.
Proposes a new method for probabilistic regression in computer vision.
problem Lack of natural probabilistic meaning in confidence-based regression.
method Energy-based model of conditional target density using deep neural network.
result 2.2% AP improvement over Faster-RCNN for object detection on COCO dataset.
Geospatial analysis lacks methods like the word vector representations and pre-trained networks that significantly boost performance across a wide range of natural language and computer vision tasks. To fill this gap, we introduce Tile2Vec, an unsupervised representation learning algorithm that extends the distribution…
Robust CLIP improves vision models' resistance to attacks.
problem Vulnerability of vision models to adversarial attacks.
method Unsupervised adversarial fine-tuning of CLIP.
result Robust CLIP prevents stealth attacks on vision models.
Proposes a new pedestrian prediction model using urban data.
problem Predicting future pedestrian behavior for safer autonomous vehicle navigation.
method Growing Hidden Markov Model (GHMM) extension using cost maps and Natural Vision principles.
result Predicts pedestrian positions more accurately over a longer horizon.
Enhances Transformer for hierarchical language understanding.
problem Lack of explicit hierarchical structure in Transformer models.
method Inspired by U-Net, integrates hierarchical processing into Transformers.
result Improved performance in chit-chat dialogue tasks.
UR-FUNNY dataset aids in understanding multimodal humor.
problem Understanding humor in a multimodal context is understudied.
method Developed a multimodal dataset (UR-FUNNY) for humor detection.
result UR-FUNNY opens the door to multimodal humor detection research.
Top 8 robotic vision systems tackled lifelong object recognition challenges.
problem Lifelong learning in robotic vision for varied, dynamic environments.
method Design of a dataset with diverse conditions and rules for evaluation.
result Robotic vision systems improved over time with dynamic object appearances.
Generative Adversarial Networks (GANs) have gathered a lot of attention from the computer vision community, yielding impressive results for image generation. Advances in the adversarial generation of natural language from noise however are not commensurate with the progress made in generating images, and still lag far …
Computer vision SSL methods show effectiveness on time series data.
problem Evaluate if computer vision SSL frameworks are effective on time series data.
method Evaluated on UCR and UEA archives, proposed a new method improving VICReg.
result Computer vision SSL frameworks can be effective on time series data.
We optimize rank-based metrics using blackbox differentiation.
problem Challenges in directly optimizing rank-based metrics due to their non-differentiable and non-decomposable nature.
method Efficient, theoretically sound, and general method for differentiating rank-based metrics with mini-batch gradient descent.
result Competitive performance on standard image retrieval datasets and improved performance on object detectors.
Recently, multiple formulations of vision problems as probabilistic inversions of generative models based on computer graphics have been proposed. However, applications to 3D perception from natural images have focused on low-dimensional latent scenes, due to challenges in both modeling and inference. Accounting for th…
We consider the problem of finding the probability that a random triangle is obtuse, which was first raised by Lewis Caroll. Our investigation leads us to a natural correspondence between plane polygons and the Grassmann manifold of 2-planes in real n-space proposed by Allen Knutson and Jean-Claude Hausmann. This cor…
Quantum hybrid vision transformers improve event classification in high energy physics.
problem Excessive computational resources for training and deploying vision transformer models.
method Constructed quantum hybrid vision transformers for high energy physics event classification.
result Quantum hybrid models achieve comparable performance to classical models with fewer parameters.
New dataset and benchmarks for lifelong robotic vision tasks.
problem Challenges in applying computer vision to robots, especially lifelong learning.
method Provided a new lifelong robotic vision dataset and benchmarks.
result Demonstrated the complexity and bottlenecks in lifelong object recognition.