This study improves GANs by learning latent distributions and pushforward maps.
problem Improving the performance of GANs with optimal transport metrics.
method Focuses on the interplay between latent distribution and generator complexity.
result Learning latent distributions and pushforward maps can significantly reduce sample complexity.
Theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.
problem Lack of precise theoretical justification for asymmetric actor-critic algorithms in reinforcement learning.
method Adapting a finite-time convergence analysis to the asymmetric actor-critic setting with linear function approximators.
result A finite-time bound reveals that the asymmetric critic eliminates aliasing errors in the agent state.
We present a generic compact computational framework relying on structured random matrices that can be applied to speed up several machine learning algorithms with almost no loss of accuracy. The applications include new fast LSH-based algorithms, efficient kernel computations via random feature maps, convex optimizati…
Since its inception, the modus operandi of multi-task learning (MTL) has been to minimize the task-wise mean of the empirical risks. We introduce a generalized loss-compositional paradigm for MTL that includes a spectrum of formulations as a subfamily. One endpoint of this spectrum is minimax MTL: a new MTL formulation…
This paper establishes a theoretical foundation for super-models via domain adaptation.
problem Reducing computational and data costs in AI for small and medium-sized enterprises.
method Two-stage diffusion process modeling, including pre-training and fine-tuning stages, using the Uhlenbeck-Ornstein process.
result The generalization error of the fine-tuning stage is dominant in domain adaptation.
Most existing binary classification methods target on the optimization of the overall classification risk and may fail to serve some real-world applications such as cancer diagnosis, where users are more concerned with the risk of misclassifying one specific class than the other. Neyman-Pearson (NP) paradigm was introd…
Diffusion models explained via cognitive science.
problem Weak sensitivity to noise family and noise level scheduling.
method Correspondence with serial reproduction in cognitive science.
result Properties of diffusion models explained by cognitive science.
This paper bridges MTL and meta-learning, showing their shared structure and efficiency.
problem Improving generalization and adaptation in multi-task and few-shot learning.
method Theoretical analysis and empirical investigation of MTL and gradient-based meta-learning.
result MTL and GBML share similar optimization formulations and predictions over unseen tasks.
When it comes to clustering nonconvex shapes, two paradigms are used to find the most suitable clustering: minimum cut and maximum density. The most popular algorithms incorporating these paradigms are Spectral Clustering and DBSCAN. Both paradigms have their pros and cons. While minimum cut clusterings are sensitive t…
Recent advances in Reinforcement Learning, grounded on combining classical theoretical results with Deep Learning paradigm, led to breakthroughs in many artificial intelligence tasks and gave birth to Deep Reinforcement Learning (DRL) as a field of research. In this work latest DRL algorithms are reviewed with a focus …
Derives a new objective to learn from human preferences without approximations.
problem Learning from human preferences through RLHF relies on approximations that can lead to pitfalls.
method Derives a new general objective ΨPO that bypasses both approximations. result Demonstrates the superiority of the new objective to Direct Preference Optimisation (DPO) empirically.
We propose a general information-theoretic approach called Seraph (SEmi-supervised metRic leArning Paradigm with Hyper-sparsity) for metric learning that does not rely upon the manifold assumption. Given the probability parameterized by a Mahalanobis distance, we maximize the entropy of that probability on labeled data…
Disagreement-based approaches generate multiple classifiers and exploit the disagreement among them with unlabeled data to improve learning performance. Co-training is a representative paradigm of them, which trains two classifiers separately on two sufficient and redundant views; while for the applications where there…
Survey on statistical theories of neural networks, focusing on approximation, training dynamics, and generative models.
problem Understanding the statistical properties and training dynamics of neural networks.
method Review of existing literature on neural networks from three perspectives: approximation, training dynamics, and generative models.
result Theoretical insights into neural network training dynamics and generative models.
This paper improves model training by using a reference model to guide target model training.
problem Improving generalization and data efficiency in model training.
method DRRho risk minimization framework based on Distributionally Robust Optimization (DRO).
result DRRho risk minimization improves generalization and data efficiency compared to training without a reference model.
Estimates rewards from a demonstrator's learning process.
problem Estimating rewards from a demonstrator's behavior.
method Leveraging the demonstrator's exploration phase for reward estimation.
result Consistent reward estimation possible without identifiability issues.
Proposes a method to adapt to new classes in a domain shift.
problem Learning new classes in a domain shift without labeled supervision.
method Inspired by prototypical networks, the method classifies target samples into shared and novel classes.
result Superior performance compared to DA and CI methods in the CIDA paradigm.
New insights into correntropy-based regression reveal robustness and unified approaches.
problem Learning robust regression functions under additive noise.
method Minimum distance estimation and conditional mean, mode, median functions.
result Unified approach to conditional mean, mode, and median functions.
This paper explores how model complexity affects performance in large language models.
problem Understanding how model complexity impacts performance in large language models.
method Employing Hidden Markov Models (HMMs) to model autoregressive LLMs and investigating the relationship between model complexity and generalization capability.
result Model complexity initially increases risk, then decreases, showcasing a 'double descent' phenomenon.
New framework uses unsupervised learning for efficient exploration in RL.
problem Efficient exploration in reinforcement learning with rich observations.
method Combines unsupervised learning and no-regret RL algorithms.
result Proves sample complexity for finding near-optimal policies.
A post-hoc framework improves model performance by calibrating different feature spaces.
problem Improving AUC performance on binary classification tasks for overconfident models.
method Identifies heterogeneous partitions of the feature space and applies post-hoc calibration techniques to each partition.
result Theoretical optimality of the framework for any model, demonstrated on deep neural networks.
New method improves language model fine-tuning without forgetting.
problem Fine-tuning language models to match specific distributions without forgetting.
method Combines Distribution Matching and Reinforcement Learning techniques.
result Adding a baseline improves constraint satisfaction, stability, and efficiency.
Study explores learning behavior of GFlowNets, revealing key mechanisms.
problem Lack of theoretical understanding of GFlowNets' learning dynamics.
method Rigorous theoretical investigation of four dimensions: convergence, sample complexity, implicit regularization, and robustness.
result Elucidates mechanisms underlying GFlowNet's learning dynamics, providing insights into performance factors.
Proposes a new approach to MSDA by introducing latent covariate shift to handle varying label distributions.
problem Challenges of conventional MSDA approaches in real-world settings where label distributions vary across domains.
method Introduces latent covariate shift (LCS) and a causal generative model with latent noises, latent content variable, and latent style variable.
result Identifies latent content variable up to block identifiability, enabling more nuanced label distribution recovery.
We introduce a novel co-learning paradigm for manifolds naturally equipped with a group action, motivated by recent developments on learning a manifold from attached fibre bundle structures. We utilize a representation theoretic mechanism that canonically associates multiple independent vector bundles over a common bas…
The paper proposes new interpretability paradigms to improve model faithfulness.
problem Improving the accuracy of explanations for complex models.
method Examining and evolving existing paradigms, proposing new models.
result Three new paradigms for interpretability are presented.
In this paper, we introduce a new online decision making paradigm that we call Thresholding Graph Bandits. The main goal is to efficiently identify a subset of arms in a multi-armed bandit problem whose means are above a specified threshold. While traditionally in such problems, the arms are assumed to be independent, …
This review explores resampling techniques for imbalanced binary classification.
problem Imbalanced classes lead to poor prediction results in classification.
method Classical, cost-sensitive, and Neyman-Pearson paradigms with resampling techniques and classification methods.
result Complex dynamics among resampling techniques, base methods, metrics, and imbalance ratios.
Unsupervised pre-training improves model generalization, but lacks theoretical understanding.
problem Lack of theoretical understanding of unsupervised pre-training's impact on model generalization.
method Introduces a novel theoretical framework to analyze and enhance generalization.
result Enhances understanding of unsupervised pre-training and fine-tuning, proposing a new regularization method.
Develops algorithms for multi-class Neyman-Pearson classification with cost sensitivity.
problem Asymmetric misclassification costs in multi-class classification problems.
method Establishes connection with cost-sensitive learning, proposes two algorithms, extends NP oracle properties.
result Proposes algorithms with theoretical guarantees for multi-class Neyman-Pearson classification.
Transfer learning has received a lot of attention in the machine learning community over the last years, and several effective algorithms have been developed. However, relatively little is known about their theoretical properties, especially in the setting of lifelong learning, where the goal is to transfer information…
Theory explains how AI models can predict unseen tasks without labeled data.
problem Understanding how AI models can generalize to unseen tasks.
method Developed a theoretical framework to analyze zero-shot prediction.
result Identified key quantities and independence relationships for generalization.
Paper analyzes inclusive KL inference using Wasserstein gradient flows.
problem Analyzing inclusive KL inference with mathematical tools.
method Gradient flows derived from PDE analysis.
result Unified view of existing sampling algorithms as inclusive-KL inference.
We propose a general information-theoretic approach called Seraph (SEmi-supervised metRic leArning Paradigm with Hyper-sparsity) for metric learning that does not rely upon the manifold assumption. Given the probability parameterized by a Mahalanobis distance, we maximize the entropy of that probability on labeled data…
In school, a teacher plays an important role in various classroom teaching patterns. Likewise to this human learning activity, the learning using privileged information (LUPI) paradigm provides additional information generated by the teacher to 'teach' learning models during the training stage. Therefore, this novel le…
Deep Sets approximates functions on sets with high-dimensional latent space.
problem Modeling functions of sets (permutation-invariant functions).
method Deep Sets, a method known to be a universal approximator for continuous set functions.
result Deep Sets' universal approximation property is only guaranteed with a sufficiently high-dimensional latent space.
A widely applied diversification paradigm is the naive diversification choice heuristic. It stipulates that an economic agent allocates equal decision weights to given choice alternatives independent of their individual characteristics. This article provides mathematically and economically sound choice theoretic founda…
Neural ODEs provide a framework for studying the training dynamics of neural networks.
problem Training dynamics of neural networks
method Dynamical mean field theory
result Derive learning curves in the high-dimensional limit
DRFLM improves federated learning by handling data heterogeneity and noise.
problem Data heterogeneity and noise in federated learning.
method Distributionally robust optimization and mixup techniques.
result Enhanced global model prediction accuracy through robust optimization and local mixup.
The 1950's foundational literature on rational mechanics exhibits two somewhat distinct paradigms to the representation of continuous distributions of defects in solids. In one paradigm, the fundamental objects are geometric structures on the body manifold, e.g., an affine connection and a Riemannian metric, which repr…
Scarcity of labeled data is a bottleneck for supervised learning models. A paradigm that has evolved for dealing with this problem is data programming. An existing data programming paradigm allows human supervision to be provided as a set of discrete labeling functions (LF) that output possibly noisy labels to input in…
The paper explores fundamental limits of learning non-hallucinating generative models.
problem Hallucinations in generative models producing invalid outputs.
method Developed a theoretical framework to analyze learnability from a learning-theoretic perspective, incorporating inductive biases.
result Non-hallucinating learning is statistically impossible without additional inductive biases.
This research compares masked diffusion models to autoregressive language models, focusing on architectural differences.
problem Comparing masked diffusion models to autoregressive language models due to architectural differences.
method Equitably compare MDMs within a decoder-only framework, investigating architectural influences.
result Decoder-only MDMs can achieve significant speedups and comparable perplexity with temperature annealing.
This work explores test-time scaling strategies for LLMs, improving sample efficiency and expressiveness.
problem Understanding the sample efficiency and expressiveness of test-time scaling strategies for LLMs.
method Established separation and expressiveness results for self-consistency, best-of-n, and self-correction strategies. result Self-correction enables Transformers to simulate online learning over multiple tasks without prior knowledge.
Games generalize the single-objective optimization paradigm by introducing different objective functions for different players. Differentiable games often proceed by simultaneous or alternating gradient updates. In machine learning, games are gaining new importance through formulations like generative adversarial netwo…
This research formalizes inductive generalization and proposes a new learning paradigm called Inductive Learning.
problem Generalization from easy to hard tasks, especially out-of-domain generalization.
method Formalizes inductive generalization, introduces Inductive Learning, and outlines steps to adapt techniques for learning model successors.
result A new learning paradigm (Inductive Learning) that emphasizes induction and universal properties of learning and computation.
Brain computer interfaces (BCI) enable direct communication with a computer, using neural activity as the control signal. This neural signal is generally chosen from a variety of well-studied electroencephalogram (EEG) signals. For a given BCI paradigm, feature extractors and classifiers are tailored to the distinct ch…
This work analyzes SGGMs, offering convergence insights and practical design tips.
problem Theoretical convergence analysis for SGGMs with a system of coupled SDEs.
method Non-asymptotic convergence analysis for three graph generation paradigms.
result Unique factors affecting convergence in SGGMs and practical hyperparameter selection.