Language grounded image understanding tasks have often been proposed as a method for evaluating progress in artificial intelligence. Ideally, these tasks should test a plethora of capabilities that integrate computer vision, reasoning, and natural language understanding. However, rather than behaving as visual Turing t…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “grounded language understanding”
Simple model outperforms neural networks on language understanding tasks.
problem Neural networks struggle with creating novel expressions from familiar ones.
method Attention-inspired modification of a baseline model, focusing on sequential thinking and acting.
result Simple model achieves good performance on gSCAN tasks, validating the benchmark.
StakeBench evaluates language understanding by linking comments to market commitments, improving model alignment with real-world outcomes.
problem Existing financial NLP benchmarks measure perceived language rather than market commitments.
method StakeBench uses observable market behavior to supervise models, testing their ability to detect commitments, identify sides, and project odds.
result Models partially recover position-side signals but struggle with later tasks, highlighting structural failures.
We are increasingly surrounded by artificially intelligent technology that takes decisions and executes actions on our behalf. This creates a pressing need for general means to communicate with, instruct and guide artificial agents, with human language the most compelling means for such communication. To achieve this i…
Zero-shot understanding of accidents from surveillance videos using vision-language models
problem Accident understanding from surveillance videos
method Three-stage pipeline with vision-language similarity, metadata-driven multi-prompt reasoning, and entropy-gated pairwise adjudicator
result Substantial improvement in harmonic-mean score over baseline
DGDS uses documents to center conversations, promising broader AI understanding.
problem DS classification by function is insufficient for complex conversations.
method Classify DS based on document grounding, analyzing classification, architecture, datasets, and models.
result DGDS can better represent current DS development trends and future AI understanding.
VALAN is a lightweight and scalable software framework for deep reinforcement learning based on the SEED RL architecture. The framework facilitates the development and evaluation of embodied agents for solving grounded language understanding tasks, such as Vision-and-Language Navigation and Vision-and-Dialog Navigation…
This paper proposes a new method to connect language and physical actions in reinforcement learning.
problem Connecting linguistic representations to the physical world in embodied agents.
method Language-conditioned goal generators to decouple sensorimotor learning from language acquisition.
result Agents can demonstrate a diversity of behaviors for any given instruction.
End-to-end KBQA system learns from multiple reasoning paths without labeled paths.
problem Lack of labeled reasoning paths limits KBQA system performance.
method End-to-end KBQA system using multiple reasoning paths.
result Demonstrates strong performance on various KBQA datasets.
Model learns image-word associations from captions using contrastive learning.
problem Phrase grounding, associating image regions to caption words.
method Optimizing word-region attention to maximize mutual information, using language model guided word substitutions for negatives.
result Model achieves 76.7% accuracy on Flickr30K Entities benchmark, a 5.7% gain from weak supervision.
Proposes a method to align language and image data.
problem Aligning language and image data for better understanding.
method Uses triplet loss to learn consistent embeddings of language and images.
result Outperforms four baselines across multiple metrics.
New approach decouples skill learning and language grounding for autonomous agents.
problem Autonomous acquisition of skills without external instructions and feedback.
method Language-Goal-Behavior (LGB) architecture with semantic representation.
result Decouples skill learning and language grounding, enabling diversity and strategy switching.
New method improves RL/IL agents' adaptability to unseen environments.
problem Current RL/IL techniques struggle with generalizing to unseen environments.
method Zero-shot compositional policy learning with multi-modal fusion and attention mechanism.
result Language grounding enhances generalization across varied environments.
Unified model learns concepts across domains like left and right.
problem Limited generalization of language concepts in inference-only models.
method Logic-Enhanced Foundation Model (LEFT) with a differentiable, domain-independent program executor.
result LEFT flexibly learns and reasons with concepts across 2D images, 3D scenes, human motions, and robotic manipulation.
Over the last few years, there has been growing interest in learning models for physically grounded language understanding tasks, such as the popular blocks world domain. These works typically view this problem as a single-step process, in which a human operator gives an instruction and an automated agent is evaluated …
Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Trainingcs.LG
New method SF-AdamW trains large models without decay phases or memory overhead.
problem Inadequate fixed compute budgets for large-scale training.
method Schedule-Free (SF) method revisited and refined.
result SF-AdamW effectively navigates loss landscape without decay phases or memory overhead.
This is a lecture note for the course DS-GA 3001 <Natural Language Understanding with Distributed Representation> at the Center for Data Science , New York University in Fall, 2015. As the name of the course suggests, this lecture note introduces readers to a neural network based approach to natural language understand…
Developing an AI economist agent using RAG, knowledge graphs, and LLMs for economic scenario analysis.
problem Economic scenario analysis using large language models and knowledge graphs.
method Proposing an RAG-based AI economist framework that utilizes knowledge graphs and LLMs.
result Improves economic coherence and traceability in generated reports.
Reinforcement learning is a promising framework for solving control problems, but its use in practical situations is hampered by the fact that reward functions are often difficult to engineer. Specifying goals and tasks for autonomous machines, such as robots, is a significant challenge: conventionally, reward function…
Survey of large language models in financial prediction and trading.
problem Improving predictability and robustness of financial predictions and trading decisions.
method Task-centered taxonomy, review of empirical evidence, design patterns, benchmarks, and challenges analysis.
result Improved predictability and robustness of financial predictions and trading decisions through large language models.
Evolutionary methods improve understanding of LLMs and their relationships.
problem Improving understanding of LLMs and their relationships.
method Relating weights to genotypes and output text to phenotypes using evolutionary methods.
result Estimated evolutionary trees reliably recover the topology of the ground-truth training tree.
Recurrent neural networks (RNNs) process input text sequentially and model the conditional transition between word tokens. In contrast, the advantages of recursive networks include that they explicitly model the compositionality and the recursive structure of natural language. However, the current recursive architectur…
CERT improves language understanding by contrastively learning sentence-level semantics.
problem Lack of sentence-level semantics in existing pretraining tasks.
method Contrastive self-supervised learning at the sentence level using back-translation augmentations.
result CERT outperforms BERT on 7 out of 11 GLUE benchmark tasks, achieving the same performance as BERT on 2 tasks.
Stories generated with neural language models have shown promise in grammatical and stylistic consistency. However, the generated stories are still lacking in common sense reasoning, e.g., they often contain sentences deprived of world knowledge. We propose a simple multi-task learning scheme to achieve quantitatively …
Migration crisis, climate change or tax havens: Global challenges need global solutions. But agreeing on a joint approach is difficult without a common ground for discussion. Public spheres are highly segmented because news are mainly produced and received on a national level. Gain- ing a global view on international d…
FUSE improves verification quality without ground truth labels.
problem Verification of model outputs using imperfect judges and reward models.
method Ensembling verifiers without ground truth labels using spectral algorithms.
result FUSE matches or improves upon semi-supervised alternatives in test-time scaling experiments.
Transformers improve logical reasoning on longer proofs but struggle with length.
problem Understanding systematic generalization in neural proof generation.
method Soft theorem-proving using Transformer models, evaluating logical consistency and inference accuracy.
result Transformers improve generalization with longer proofs but have difficulty with length.
Humor is a unique and creative communicative behavior displayed during social interactions. It is produced in a multimodal manner, through the usage of words (text), gestures (vision) and prosodic cues (acoustic). Understanding humor from these three modalities falls within boundaries of multimodal language; a recent r…
lamBERT learns language and actions using multimodal BERT.
problem Learning language and actions in complex environments.
method Extending BERT to multimodal representation and integrating with reinforcement learning.
result lamBERT model achieved higher rewards in multitask and transfer settings.
The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?cs.CL
Language model benchmarks often misrepresent true understanding, revealing vulnerabilities in evaluation methods.
problem Language model benchmarks fail to accurately reflect true language understanding and adaptability.
method Systematic analysis of NLP evaluation frameworks, identifying vulnerabilities in static benchmarks, human evaluation protocols, and LLM-as-judge frameworks.
result Current evaluation methods are unreliable and need improvement to accurately assess LLM performance.
ZeroSCROLLS benchmarks zero-shot natural language understanding over long texts.
problem Evaluate natural language understanding models over long texts without training data.
method Adapt six tasks from SCROLLS benchmark and add four new datasets, including novel aggregation tasks.
result Claude outperforms ChatGPT, and GPT-4 achieves highest average score.
Optimus pre-trains sentences in a latent space for various NLP tasks.
problem Training large-scale language models for diverse NLP tasks.
method Pre-trained Variational Autoencoder (VAE) on large text corpus, fine-tuned for various tasks.
result Optimus achieves state-of-the-art on VAE language modeling benchmarks.
The study categorizes reward errors in reinforcement learning, finding some can be beneficial.
problem Training language models with imperfect proxy rewards.
method Theoretical analysis of policy gradient optimization and categorization of reward errors.
result Reward errors can be benign or even beneficial, preventing policy from stalling.
Word2Vec (W2V) and GloVe are popular, fast and efficient word embedding algorithms. Their embeddings are widely used and perform well on a variety of natural language processing tasks. Moreover, W2V has recently been adopted in the field of graph embedding, where it underpins several leading algorithms. However, despit…
Autonomous reinforcement learning agents, like children, do not have access to predefined goals and reward functions. They must discover potential goals, learn their own reward functions and engage in their own learning trajectory. Children, however, benefit from exposure to language, helping to organize and mediate th…
Programmable Agentscs.AI
We build deep RL agents that execute declarative programs expressed in formal language. The agents learn to ground the terms in this language in their environment, and can generalize their behavior at test time to execute new programs that refer to objects that were not referenced during training. The agents develop di…
Genomic models learn DNA sequences to predict functions.
problem Understanding complex genetic interactions.
method Training LLMs on DNA sequences to predict functions.
result gLMs can predict functions of DNA elements.
Paper investigates multimodal contrastive learning and incorporates unpaired data.
problem Improving feature learning ability of multimodal models under noisy data.
method Initiates investigation of nonlinear loss functions for multimodal contrastive learning, analyzes performance, proposes new loss incorporating unpaired data.
result MMCL can outperform unimodal contrastive learning and robustly handle noisy data.
This paper investigates teacher hacking during language model distillation and proposes methods to mitigate it.
problem Teacher hacking during language model distillation, leading to suboptimal performance.
method A controlled experimental setup involving an oracle LM, teacher LM, and student LM, using fixed offline or online data generation techniques.
result Data diversity is the key factor in preventing teacher hacking during distillation.
TIM framework uses LLMs and domain experts to infer DeFi user transaction intents.
problem Challenges in understanding user intent in DeFi transactions due to complex interactions and opaque logs.
method TIM framework leverages a DeFi intent taxonomy, multi-agent LLM system, and a Meta-Level Planner.
result TIM significantly outperforms existing methods in inferring user transaction intents.
Martingale Doppelgänger-Eval benchmarks VLMs on candlestick evidence vs. trend extrapolation
problem Auditing whether VLMs use chart evidence or trend extrapolation
method Proving formal limitations and designing controlled mechanisms
result Identifying regression coefficients for evidence vs. trend
LR-Robot automates SLRs with AI, expert oversight, and multidimensional analysis.
problem Efficient but contextually limited outputs from existing SLR frameworks.
method Human-in-the-loop process, structured knowledge sources, retrieval-augmented generation.
result Empirical demonstration of AI-driven literature synthesis in option pricing.
Unified approach to RLHF tackles uncertainty in reward function.
problem Uncertainty in reward function learned from human feedback.
method Value-incentivized preference optimization (VPO) that regularizes the reward function with value function.
result Theoretical and practical guarantees for both online and offline RLHF settings.
A new framework evaluates LLMs by considering judge reliability.
problem Evaluating LLMs without ground truth labels can lead to biased results.
method Introduces judge-specific discrimination parameters and estimates model quality and judge reliability.
result Improves agreement with human preferences and produces calibrated uncertainty quantification.
We address the problem of bootstrapping language acquisition for an artificial system similarly to what is observed in experiments with human infants. Our method works by associating meanings to words in manipulation tasks, as a robot interacts with objects and listens to verbal descriptions of the interactions. The mo…
Continuous vector representations of words and objects appear to carry surprisingly rich semantic content. In this paper, we advance both the conceptual and theoretical understanding of word embeddings in three ways. First, we ground embeddings in semantic spaces studied in cognitive-psychometric literature and introdu…
Although deep convolutional networks have achieved improved performance in many natural language tasks, they have been treated as black boxes because they are difficult to interpret. Especially, little is known about how they represent language in their intermediate layers. In an attempt to understand the representatio…
TextHide secures language understanding tasks by adding minimal encryption.
problem Mitigating privacy risks in natural language understanding tasks.
method TextHide adds a simple encryption step to prevent data recovery.
result TextHide reduces accuracy loss by only 1.9% on GLUE benchmark.