The paper uses geometry to understand how neural networks learn.
problem Understanding the learning capability of neural networks.
method Statistical and differential geometric analysis of neural networks performing simple regression.
result Neural networks with higher generalization capability have a slower convergence rate.
New framework transfers latent knowledge from weak to strong models.
problem Aligning superhuman LLMs with human feedback.
method Transfer learning framework using refinement approach.
result Proves weak-to-strong generalization is possible.
The study uncovers latent capabilities of language models via causal representation learning.
problem Rigorous causal evaluations of language model capabilities are challenging due to confounding effects and computational costs.
method Proposes a causal representation learning framework to identify latent capability factors as causally interrelated after controlling for a common confounder (base model).
result Identifies a three-node linear causal structure explaining performance variations across 1500 models and six benchmarks.
DNNs generalize object recognition in novel orientations via neurons tuned to common features.
problem Understanding how DNNs generalize to objects in novel orientations.
method Training DNNs with familiar objects from multiple viewpoints and analyzing neuron responses.
result DNNs disseminate orientation-invariance from familiar objects to recognize objects in novel orientations.
ML Compass helps organizations choose AI models that balance utility, cost, and compliance.
problem Selecting AI models that meet user utility, deployment costs, and compliance requirements.
method Develops ML Compass, a framework for constrained optimization over a capability-cost frontier, using internal measures and empirical data.
result ML Compass produces deployment-aware recommendations that differ from capability-only rankings, clarifying trade-offs between capability, cost, and safety.
Fine-tuning LLMs improves capability but harms safety, study finds.
problem Balancing capability and safety in LLM fine-tuning.
method Theoretical framework and numerical experiments for two safety-aware fine-tuning strategies.
result Characterization of fundamental limits of safety-capability trade-off in LLM fine-tuning.
The paper tackles extrapolation in generative models by enforcing independence of mechanisms.
problem How to make generative models extrapolate to new, unseen environments?
method Developed a theoretical framework for independence of mechanisms, demonstrated on toy examples and real-world data.
result Extrapolation capabilities of generative models can be improved by enforcing independence of mechanisms explicitly during training.
Reward hacking exploits misspecified rewards, affecting agent capabilities and true performance.
problem Reward hacking in RL models exploiting reward misspecifications.
method Constructed four RL environments with misspecified rewards; analyzed agent capabilities and behavior.
result More capable agents exploit reward misspecifications, achieving higher proxy reward but lower true reward.
SQWA improves low-precision DNNs with model averaging and quantization.
problem Designing good generalization DNNs with quantized weights.
method Floating-point model training, direct quantization, multiple low-precision models, weight averaging, re-quantization, fine-tuning, loss visualization.
result State-of-the-art results for 2-bit QDNNs on CIFAR-100 and ImageNet datasets.
Wind energy forecasting helps to manage power production, and hence, reduces energy cost. Deep Neural Networks (DNN) mimics hierarchical learning in the human brain and thus possesses hierarchical, distributed, and multi-task learning capabilities. Based on aforementioned characteristics, we report Deep Belief Network …
Performing supervised learning from the data synthesized by using Generative Adversarial Networks (GANs), dubbed GAN-synthetic data, has two important applications. First, GANs may generate more labeled training data, which may help improve classification accuracy. Second, in scenarios where real data cannot be release…
SYNTHONY selects tabular synthesizers based on stress profiling and user intent.
problem Non-uniform performance of tabular generative models across datasets.
method Stress profiling and intent-conditioned tabular synthesis selection.
result Meta-features predict synthesizer performance, improving selection accuracy.
Layout design with complex constraints is a challenging problem to solve due to the non-uniqueness of the solution and the difficulties in incorporating the constraints into the conventional optimization-based methods. In this paper, we propose a design method based on the recently developed machine learning technique,…
RLVR maintains safety while improving reasoning capabilities in LLMs.
problem Safety-capability tradeoff in fine-tuning LLMs.
method Reinforcement Learning with Verifiable Rewards (RLVR) and theoretical analysis.
result RLVR can enhance reasoning while maintaining safety guardrails.
A meta-learning method learns adaptive robust loss functions for noisy labels.
problem Handling robust learning with noisy labels and optimizing hyperparameters.
method Adaptive learning of robust loss hyperparameters through mutual improvement with network parameters.
result Generalized and effective robust loss functions with good generalization capability.
In this work, we address the problem of modifying textual attributes of sentences. Given an input sentence and a set of attribute labels, we attempt to generate sentences that are compatible with the conditioning information. To ensure that the model generates content compatible sentences, we introduce a reconstruction…
Paper examines LLM capability benchmarks through construct validity, favoring nomological account.
problem Linking theoretical capabilities to empirical measurements in LLMs.
method Contrasts three frameworks: nomological, inferential, and causal.
result Nomological account provides best foundation for LLM research.
We evaluate the distribution learning capabilities of generative adversarial networks by testing them on synthetic datasets. The datasets include common distributions of points in Rn space and images containing polygons of various shapes and sizes. We find that by and large GANs fail to faithfully recreate point dat…
Recursive KalmanNet generalizes well in noisy, out-of-distribution scenarios.
problem Generalization in noisy, out-of-distribution scenarios.
method Recurrent neural network guided by a Kalman filter.
result Recursive KalmanNet performs well in scenarios with different temporal dynamics from training data.
A new method learns complex dynamical systems from data efficiently.
problem Learning complex dynamical systems from large-scale data efficiently.
method Low-rank structured variational autoencoding framework for nonlinear Gaussian state-space models.
result Consistently demonstrates better predictive capabilities compared to other models.
Music Inpainting is the task of filling in missing or lost information in a piece of music. We investigate this task from an interactive music creation perspective. To this end, a novel deep learning-based approach for musical score inpainting is proposed. The designed model takes both past and future musical context i…
The study explores the strengths and weaknesses of models that generalize from weak to strong supervision.
problem Understanding the limitations and capabilities of models that generalize from weak to strong supervision.
method Theoretical analysis and experimental validation in both classification and regression settings.
result Theoretical bounds reveal the importance of strong generalization and calibration of the weak model and a careful balance in the training process.
Much of the analysis of economic growth has focused on the study of aggregate output. Here, we deviate from this tradition and look instead at the structure of output embodied in the network connecting countries to the products that they export.We characterize this network using four structural features: the negative r…
Agents need world models to generalize multi-step tasks.
problem The necessity of world models for flexible, goal-directed behavior.
method Formal analysis and demonstration of the necessity of world models for agents to generalize multi-step tasks.
result World models are necessary for agents to generalize to multi-step goal-directed tasks.
Chronos-2 forecasts multivariate and covariate data without task-specific training.
problem Limited applicability of existing time series forecasting models to real-world multivariate and covariate data.
method Chronos-2 uses a group attention mechanism for in-context learning across multiple time series.
result Chronos-2 achieves state-of-the-art performance across comprehensive benchmarks.
A new framework ensures model safety by retaining old model capabilities while improving new tasks.
problem Catastrophic forgetting in continual learning systems.
method Retention-centric framework with data-dependent constraints and an efficient constrained optimization algorithm.
result The method ensures that new models retain important capabilities of old models while improving new tasks.
We present Manifold Alignment Determination (MAD), an algorithm for learning alignments between data points from multiple views or modalities. The approach is capable of learning correspondences between views as well as correspondences between individual data-points. The proposed method requires only a few aligned exam…
Paper proposes a new financial fraud detection system using improved RF and GBM.
problem Detecting financial fraud with high accuracy and efficiency.
method Introduces GBM-SSRF, a novel model combining GBM and SSRF.
result GBM-SSRF model outperforms traditional methods in fraud detection.
We present an agent based model of a single asset financial market that is capable of replicating several non-trivial statistical properties observed in real financial markets, generically referred to as stylized facts. While previous models reported in the literature are also capable of replicating some of these stati…
New approach extracts AI model representations for steering and monitoring.
problem Understanding and steering AI models' internal knowledge representation.
method Feature learning advances for scalable linear concept representations.
result Concept representations enable steering and monitoring of AI models.
DAEs can generate images without additional loss terms, inheriting VAE properties.
problem Difficulty in using VAEs for practical generative modelling.
method Empirical exploration of DAEs for image generation without novel methods.
result DAEs can generate images successfully without additional loss terms.
Proposes CLRS-Text, a new benchmark for evaluating LM reasoning capabilities.
problem Lack of transferable benchmarks for evaluating reasoning capabilities of language models.
method Developed a textual version of the CLRS benchmark, generating diverse algorithmic tasks.
result Demonstrates a novel challenge for the LM reasoning community and validates prior work.
The Interaction-Transformation (IT) is a new representation for Symbolic Regression that restricts the search space into simpler, but expressive, function forms. This representation has the advantage of creating a smoother search space unlike the space generated by Expression Trees, the common representation used in Ge…
Predicts language model performance from public models without training.
problem Understanding how language model performance varies with scale.
method Observational approach building scaling laws from publicly available models.
result Smooth, sigmoidal behavior of emergent phenomena is predictable from small models.
Study shows statistical biases can mislead transformer models, impairing their generalization.
problem Statistical biases in transformers affect their ability to generalize.
method Evaluated transformer models on synthetic algorithmic tasks with varying statistical biases.
result Statistical biases lead to overestimation of transformer models' generalization capabilities.
SrvfNet aligns multiple functional data to templates without supervision.
problem Aligning large collections of functional data to templates without labeled data.
method Generative deep learning framework using SRVF and fully-connected layers.
result Framework achieves alignment and optimal template prediction without supervision.
Study shows how a strong model can learn a task's feature while retaining other capabilities.
problem How to align superhuman AI systems using weak-to-strong generalization.
method Two-layer neural networks, reward-model learning, multi-step SGD, feature learning.
result The strong model efficiently learns task features while retaining general capabilities.
BIG-bench benchmarks language models, revealing their strengths and weaknesses.
problem Understanding and quantifying the capabilities of large language models.
method Developed the BIG-bench benchmark with 204 tasks from various domains, evaluated across multiple model sizes and types.
result Model performance improves with scale but remains poor, with breakthrough behaviors often involving multiple steps.
When building a unified vision system or gradually adding new capabilities to a system, the usual assumption is that training data for all tasks is always available. However, as the number of tasks grows, storing and retraining on such data becomes infeasible. A new problem arises where we add new capabilities to a Con…
Paper explores anchoring for vision models, improving generalization and safety.
problem Anchoring can lead to undesirable shortcuts, limiting generalization.
method Introduced a new anchored training protocol with a regularizer to mitigate undesirable shortcuts.
result Significant performance gains in generalization and safety metrics.
Variational autoencoders (VAEs) are widely used deep generative models capable of learning unsupervised latent representations of data. Such representations are often difficult to interpret or control. We consider the problem of unsupervised learning of features correlated to specific labels in a dataset. We propose a …
We present in this paper a model for forecasting short-term power loads based on deep residual networks. The proposed model is able to integrate domain knowledge and researchers' understanding of the task by virtue of different neural network building blocks. Specifically, a modified deep residual network is formulated…
Deep Reinforcement Learning (DRL) has emerged as a powerful control technique in robotic science. In contrast to control theory, DRL is more robust in the thorough exploration of the environment. This capability of DRL generates more human-like behaviour and intelligence when applied to the robots. To explore this capa…
New method predicts language model scaling risks with limited data.
problem Accurately predicting model capabilities and risks with limited data.
method Introduces a robust estimation framework using beta-binomial distribution and dynamic sampling strategy.
result More accurate predictions of rare risks and capabilities at lower computational cost.
Research into automated systems for detecting and classifying marine mammals in acoustic recordings is expanding internationally due to the necessity to analyze large collections of data for conservation purposes. In this work, we present a Convolutional Neural Network that is capable of classifying the vocalizations o…
RNNs struggle with in-context retrieval, while Transformers excel.
problem In-context retrieval capability of RNNs.
method Theoretical analysis and experimental techniques (CoT, RAG, Transformer layer).
result Enhancing RNNs with techniques improves their in-context retrieval capability, closing the representation gap with Transformers.
Process capability index (PCI) is a commonly used statistic to measure ability of a process to operate within the given specifications or to produce products which meet the required quality specifications. PCI can be univariate or multivariate depending upon the number of process specifications or quality characteristi…
Deep reinforcement learning algorithms have shown an impressive ability to learn complex control policies in high-dimensional tasks. However, despite the ever-increasing performance on popular benchmarks, policies learned by deep reinforcement learning algorithms can struggle to generalize when evaluated in remarkably …