Deep learning models predict generalization gaps without specific task or architecture.
problem Predicting when deep learning works across different tasks and architectures.
method Created a dataset of 13,500 neural networks trained on various spiral datasets and parameters. Used this dataset to train predictors for generalization gaps.
result DNNs and RNNs outperform linear models in predicting generalization gaps, with RNNs achieving R2=0.584. DI-SVM improves brain condition decoding performance via domain independence.
problem Transfer learning in brain imaging data with large p and small n.
method DI-SVM minimizes domain dependence via HSIC to learn common features.
result DI-SVM outperforms eight competing methods on brain decoding tasks.
Framework learns portable representations for diverse tasks.
problem Creating task-independent abstract representations for diverse environments.
method Autonomously learns portable representations in egocentric space.
result Portable representations enable task-independent planning and transfer.
We discuss a general method to learn data representations from multiple tasks. We provide a justification for this method in both settings of multitask learning and learning-to-learn. The method is illustrated in detail in the special case of linear feature learning. Conditions on the theoretical advantage offered by m…
Paper proposes a new graph neural network for task-independent graph embeddings.
problem Task-dependent embeddings in graph learning domains.
method Incorporates a novel graph neural network and rich Graph Kernels for unsupervised and supervised learning.
result Consistently outperforms existing models by 3% - 8% on graph classification benchmarks.
New test detects independence in streaming data, adapting to data complexity.
problem Independence testing in streaming data with adaptive stopping.
method Sequential kernelized independence tests using betting principles.
result Valid inference in streaming data with improved power.
Testing (conditional) independence of multivariate random variables is a task central to statistical inference and modelling in general - though unfortunately one for which to date there does not exist a practicable workflow. State-of-art workflows suffer from the need for heuristic or subjective manual choices, high c…
Theory explains how AI models can predict unseen tasks without labeled data.
problem Understanding how AI models can generalize to unseen tasks.
method Developed a theoretical framework to analyze zero-shot prediction.
result Identified key quantities and independence relationships for generalization.
New ITID assumption improves generalization in practical tasks.
problem Generalization theories fail to interpret many generalization phenomena and guide practical learning tasks due to the IID assumption.
method Proposes ITID assumption to consider task properties, derives a new generalization bound, and introduces an invariance enhancement algorithm.
result Demonstrates the effectiveness of the new generalization theory in improving practical generalization performance.
New framework for task-independent legged locomotion.
problem Building stable legged locomotion systems in robotics.
method Task-independent spiking central pattern generator using learning methods.
result Robotic legged locomotion at different speeds and within the same gait cycle.
We present a multi-task learning approach to jointly estimate the means of multiple independent data sets. The proposed multi-task averaging (MTA) algorithm results in a convex combination of the single-task maximum likelihood estimates. We derive the optimal minimum risk estimator and the minimax estimator, and show t…
Paper proposes an ensemble model for writer-independent offline signature verification using deep learning.
problem Difficulty in distinguishing genuine signatures from skilled forgeries in writer-independent offline signature verification.
method Used an ensemble model with two CNNs for feature extraction, RGBT for classification, and stacking for final prediction.
result Achieved state-of-the-art performance on various datasets.
Parallel score matching accelerates DPM training and improves density estimation.
problem Extended training periods and limited modeling flexibility in DPMs.
method Partitioning the learning task into independent time sub-intervals and modeling the score at each time point separately.
result Significant acceleration of training process and improved density estimation performance.
New algorithms find half-optimal independent sets in sparse graphs.
problem Finding large independent sets in sparse random graphs.
method Low-degree polynomial algorithms.
result Low-degree polynomial algorithms can find independent sets of half-optimal size.
The paper shows how the timing of prediction impacts model performance in healthcare.
problem The timing of prediction affects model performance in healthcare.
method The paper compares two prediction schemes: outcome-dependent and outcome-independent.
result An outcome-independent scheme outperforms an outcome-dependent scheme.
HSIC loss improves robust regression and classification models.
problem Learning robust models in the absence of target domain data.
method Adapted HSIC loss for unsupervised covariate shift.
result Models outperform standard methods on covariate shift tasks.
We propose a streaming algorithm for the binary classification of data based on crowdsourcing. The algorithm learns the competence of each labeller by comparing her labels to those of other labellers on the same tasks and uses this information to minimize the prediction error rate on each task. We provide performance g…
Paper introduces a flow-based framework for representation learning.
problem Capturing fine-grained structural details in complex data distributions.
method Zero-flow criterion for conditional independence and a tractable loss function.
result The zero-flow criterion enables learning of sufficient information from data.
Study evaluates natural language models' ability to generalize across tasks.
problem Natural language models struggle with generalizing to new tasks.
method Empirical evaluation of state-of-the-art models using new metrics.
result Models require extensive in-domain training and are prone to forgetting.
This work explores the connection between distances and kernels for conditional independence.
problem Measuring conditional independence in various fields like causal discovery and feature selection.
method Investigates the relationship between conditional independence measures induced by distances and reproducing kernels.
result Some kernel-based conditional independence measures are not equivalent to distance-based measures.
DiffusionBlocks trains neural networks by breaking them into independent blocks, reducing memory usage.
problem Memory bottlenecks in end-to-end neural network training.
method Transforming transformer-based networks into independent trainable blocks via a denoising process.
result Independent block-wise training matches end-to-end training performance while reducing memory requirements.
Neurosymbolic predictors fail to model uncertainty under independence assumption.
problem Neurosymbolic predictors' reliance on independence assumption limits their ability to model uncertainty.
method Formal analysis of NeSy predictors under independence assumption.
result Assuming independence among symbolic concepts prevents NeSy predictors from representing uncertainty.
Meta-learning bound uses conditional mutual information.
problem Bounding generalization performance in meta-learning.
method Extends CMI framework to meta-learning with a meta-supersample.
result Explicit bound involving two CMI terms.
Identifying dependency in multivariate data is a common inference task that arises in numerous applications. However, existing nonparametric independence tests typically require computation that scales at least quadratically with the sample size, making it difficult to apply them to massive data. Moreover, resampling i…
RIMs improve generalization by specializing modular structures.
problem Improving generalization and robustness to changes in tasks.
method Recurrent Independent Mechanisms (RIMs) architecture with independent dynamics, sparing communication, and selective updates.
result RIMs lead to dramatic improvement in generalization on tasks with varying factors.
Representing distributions over permutations can be a daunting task due to the fact that the number of permutations of n objects scales factorially in n. One recent way that has been used to reduce storage complexity has been to exploit probabilistic independence, but as we argue, full independence assumptions impo…
New unsupervised learning technique learns independent kernels for better machine learning tasks.
problem Improving unsupervised representation learning for machine learning tasks.
method Stacking convolutional transforms using alternating proximal minimization scheme.
result DCTL outperforms shallow version CTL on benchmark datasets.
A Closer Look at Disentangling in β-VAE shows non-monotonic inference performance.
problem Learning disentangled representations from data.
method Generalization of VAE using variational inference with hyperparameter β.
result Non-monotonic inference performance in β-VAE with a finite optimal β.
LCIT tests conditional independence using latent representations.
problem Detecting conditional independencies in statistical and machine learning tasks.
method Generative framework for learning latent representations of target variables X and Y, then testing for remaining dependencies.
result LCIT outperforms state-of-the-art baselines consistently under different metrics and settings.
New research shows Multi-Task Learning does not behave like Multi-Objective Optimization.
problem Applying MOO methods to MTL problems is not equivalent.
method Comparing MTL and MOO approaches on Multi-Fashion-MNIST datasets.
result A single model can perform as well as optimizing multiple objectives.
New method tests causal relationships from data without needing to learn the entire graph.
problem Testing if a causal graph belongs to a specific Markov equivalence class from observational data.
method Established bounds on the number of independence tests required and provided an algorithm that matches these bounds.
result Testing requires exponentially less independence tests compared to learning, especially in graphs with high in-degrees and small clique sizes.
This research explores a modified VAE model to learn disentangled representations for object recognition.
problem Learning invariant representations for object recognition from diverse appearances.
method Develops a modified Variational Autoencoder (β-VAE) to enforce disentangled representations using variational inference. result Demonstrates that the incompatibility between β-VAE's conditional independence and latent variable independence leads to non-monotonic inference performance. Low-rank framework for task-specific LLM ranking from sparse comparisons.
problem Challenges in reliable task-specific ranking of LLMs under sparse, imbalanced comparisons.
method Low-rank modeling of task-by-model ability matrix, max-norm accurate estimator, task-wise top-K recovery guarantees, uncertainty quantification framework.
result Improves sample efficiency and produces tighter, better-calibrated ranking certificates.
Although highly correlated, speech and speaker recognition have been regarded as two independent tasks and studied by two communities. This is certainly not the way that people behave: we decipher both speech content and speaker traits at the same time. This paper presents a unified model to perform speech and speaker …
It is often stated in papers tackling the task of inferring Bayesian network structures from data that there are these two distinct approaches: (i) Apply conditional independence tests when testing for the presence or otherwise of edges; (ii) Search the model space using a scoring metric. Here I argue that for complete…
New method uses adversarial training for blind source separation.
problem Blind single-channel source separation challenge.
method Adversarial training for independence of sources.
result Good performance validated on image sources.
CoNN uses cooperative neural networks to leverage prior independence structure for improved text classification.
problem Improving text classification accuracy by exploiting prior independence structure.
method CoNN employs a set of cooperatively trained neural networks to capture latent representations based on prior independence structure.
result Demonstrated a 23% reduction in error on the MultiSent dataset compared to state-of-the-art methods.
We introduce the blind subspace deconvolution (BSSD) problem, which is the extension of both the blind source deconvolution (BSD) and the independent subspace analysis (ISA) tasks. We examine the case of the undercomplete BSSD (uBSSD). Applying temporal concatenation we reduce this problem to ISA. The associated `high …
Reliable measures of statistical dependence could be useful tools for learning independent features and performing tasks like source separation using Independent Component Analysis (ICA). Unfortunately, many of such measures, like the mutual information, are hard to estimate and optimize directly. We propose to learn i…
Sequential Kernel-based Conditional Independence Testing via Adaptive Betting
problem Testing conditional independence
method Testing-by-betting on an adaptively optimized Kernel Conditional Independence statistic
result Significantly reduces Type I error inflation while preserving high power
Conditional independence tests (CI tests) have received special attention lately in Machine Learning and Computational Intelligence related literature as an important indicator of the relationship among the variables used by their models. In the field of Probabilistic Graphical Models (PGM)--which includes Bayesian Net…
Neurons in higher cortical areas, such as the prefrontal cortex, are known to be tuned to a variety of sensory and motor variables. The resulting diversity of neural tuning often obscures the represented information. Here we introduce a novel dimensionality reduction technique, demixed principal component analysis (dPC…
We compute the transition probability between two learning tasks, and show that it decomposes into two factors. The first depends on the geometry of the loss landscape of a model trained on each task, independent of any particular model used. This is related to an information theoretic distance function, but is insuffi…
Transferring knowledge across a sequence of reinforcement-learning tasks is challenging, and has a number of important applications. Though there is encouraging empirical evidence that transfer can improve performance in subsequent reinforcement-learning tasks, there has been very little theoretical analysis. In this p…
SRPO improves AI alignment with human preferences through self-improvement and task-independent optimization.
problem AI models trained with RLHF lack self-correction mechanisms and struggle with task generalization.
method SRPO formulates the preference learning problem as a min-max objective, optimizing a self-improvement policy and a generative policy in an adversarial fashion, making the solution task-independent.
result SRPO outperforms existing methods, achieving 90% AI Win-Rate on XSum and 56% on Arena-Hard prompts after a single revision.
New algorithms test independence with fewer samples by using predictive information.
problem Testing independence of distributions with limited samples.
method Augmented distribution testing framework that incorporates predictive information.
result Optimal sample complexity achieved, matching lower bounds.
FSN model improves few-shot learning by generalizing to new tasks.
problem Few-shot learning struggles with tasks outside its training domain.
method FSN uses topology-inspired approach to model classes flexibly.
result FSN outperforms state-of-the-art models on new tasks.
OT-ICA uses optimal transport to find independent components, outperforming traditional methods.
problem Finding independent components from linear mixtures of signals.
method OT-ICA uses the squared Wasserstein distance to maximize non-Gaussianity, optimizing projections via gradient descent.
result OT-ICA outperforms traditional proxy-based methods in various applications.