Study on discrepancy principle for learning algorithms in nonparametric regression.
problem Determining optimal iteration number in nonparametric regression with unknown optimal iteration.
method Investigates discrepancy principle and modified principles for kernelized spectral filters, using deviation inequalities and change-of-norm arguments.
result Classical discrepancy principle is adaptive for slow rates, while modified principles are adaptive for faster rates.
We present a new statistical learning paradigm for Boltzmann machines based on a new inference principle we have proposed: the latent maximum entropy principle (LME). LME is different both from Jaynes maximum entropy principle and from standard maximum likelihood estimation.We demonstrate the LME principle BY deriving …
Deeper neural networks learn lower frequency functions faster, according to a new principle.
problem Understanding why deeper learning is faster.
method Fourier analysis and filtering method to separate and analyze the frequency distribution of neural network outputs.
result Deeper hidden layers in neural networks bias towards lower frequency functions during training.
New method prevents forgetting in learning new tasks.
problem Poor ability of models to solve new problems without forgetting.
method Task-agnostic hierarchical information-theoretic optimality principle with Mixture-of-Variational-Experts layer.
result Demonstrated competitive performance in continual supervised and reinforcement learning.
Deep learning improves option pricing in incomplete markets.
problem Optimal pricing and hedging in incomplete jump diffusion markets.
method Stackelberg game approach, deep learning (feedforward and LSTM networks).
result Deep learning algorithm outperforms traditional methods in incomplete markets.
PRI-VAE learns disentangled representations by optimizing principle-of-relevant-information.
problem Learning disentangled representations under VAE framework remains unknown.
method Proposes PRI-VAE, a novel learning objective to optimize disentanglement.
result Demonstrates effectiveness of PRI-VAE on four benchmark datasets.
How do organisms recognize their environment by acquiring knowledge about the world, and what actions do they take based on this knowledge? This article examines hypotheses about organisms' adaptation to the environment from machine learning, information-theoretic, and thermodynamic perspectives. We start with construc…
New principles needed for scaling large language models, challenging traditional regularization methods.
problem The shift from generalization to scaling in machine learning requires new guiding principles.
method Examining the effectiveness of traditional regularization methods in the scaling-centric era.
result Traditional principles of regularization may not generalize to larger scales, highlighting new phenomena like scaling law crossover.
Unified framework explains all types of learning, including brain.
problem Lack of clear explanation for deep learning success.
method Constructing a learning principle that equates all learning to probability estimation.
result Unified understanding of learning across different fields.
The paper proposes a principle for dynamically adjusting the granularity of reinforcement learning abstractions.
problem Lack of general principles for dynamically adjusting the granularity of reinforcement learning abstractions.
method The paper proposes a principle based on rate-distortion theory, formalized through a performance certificate decomposing value error into learning and abstraction error bounds.
result Soft state-action abstractions can achieve near-optimal performance under substantial lossy compression of state and action information.
Paper proposes a compression principle for neural networks using Bayesian optimization.
problem Finding methods for making generalizable predictions in machine learning.
method Compression principle and Bayesian optimization approach.
result Optimal predictive models minimize total compressed message length of data and model definition.
Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, some empirical studies of DNNs reported a universal phenomenon of Frequency Principle (F-Principle): a DNN tends to learn a target function from low to high frequencies during the training. The F-Principle has been very use…
Discussing AI's difficulty and physics' simplicity, suggesting AI benefits from physics principles.
problem AI difficulty compared to physics simplicity.
method Drawing on physical intuition and theoretical physics to improve AI.
result AI and physics principles are strongly coupled through sparsity.
Information-theoretic bounded rationality describes utility-optimizing decision-makers whose limited information-processing capabilities are formalized by information constraints. One of the consequences of bounded rationality is that resource-limited decision-makers can join together to solve decision-making problems …
Use simplified layerwise linear models to understand neural dynamics.
problem Complex neural network dynamics are hard to grasp.
method Apply simplified layerwise linear models to explain neural phenomena.
result Simplified models explain neural collapse, emergence, etc.
The minimum description length (MDL) principle in supervised learning is studied. One of the most important theories for the MDL principle is Barron and Cover's theory (BC theory), which gives a mathematical justification of the MDL principle. The original BC theory, however, can be applied to supervised learning only …
Previous studies have shown that deep neural networks (DNNs) with common settings often capture target functions from low to high frequency, which is called Frequency Principle (F-Principle). It has also been shown that F-Principle can provide an understanding to the often observed good generalization ability of DNNs. …
New principle optimizes bandit decisions with context.
problem Dealing with general function classes and large context spaces in bandits.
method Upper Counterfactual Confidence Bounds (UCCB) principle.
result Proves optimality and efficiency in complex settings.
Bayesian methods promise to fix many shortcomings of deep learning, but they are impractical and rarely match the performance of standard methods, let alone improve them. In this paper, we demonstrate practical training of deep networks with natural-gradient variational inference. By applying techniques such as batch n…
SDAMI enhances interpretable high-dimensional regression with sparse deep learning and footprint principle.
problem Personalized models for small samples and high-dimensional features with interpretability.
method Sparse Deep Additive Model with Interactions (SDAMI) combining sparsity-driven feature selection and deep subnetworks.
result SDAMI successfully identifies pure interactions with near-zero false positive rates.
It remains a puzzle that why deep neural networks (DNNs), with more parameters than samples, often generalize well. An attempt of understanding this puzzle is to discover implicit biases underlying the training process of DNNs, such as the Frequency Principle (F-Principle), i.e., DNNs often fit target functions from lo…
Paper proves large deviation principle for stochastic approximations.
problem Asymptotic estimates of learning algorithm deviations.
method Weak convergence approach to large deviations.
result Identifies appropriate scaling sequence and new representation for rate function.
QPLEX learns efficient multi-agent Q-values by enforcing IGM principle.
problem Scalable multi-agent reinforcement learning with IGM consistency.
method Dueling duplex network architecture to enforce IGM principle.
result QPLEX achieves high sample efficiency and benefits from offline data.
Data science principles enhance AI interpretability for better user control.
problem Risks from opaque AI models without clear impacts.
method Synthesizes principles from interpretability literature, emphasizing audience goals.
result Illustrates basic techniques and criteria for evaluating interpretability.
New algorithm solves online resource allocation problems efficiently.
problem Dynamic resource allocation in operations research.
method Minimal Selection Principle and MSoE algorithm.
result Ensures optimal cumulative regret bounds in dynamic resource allocation.
Machine Learning algorithms are typically regarded as appropriate optimization schemes for minimizing risk functions that are constructed on the training set, which conveys statistical flavor to the corresponding learning problem. When the focus is shifted on perception, which is inherently interwound with time, recent…
We propose a new active learning by query synthesis approach using Generative Adversarial Networks (GAN). Different from regular active learning, the resulting algorithm adaptively synthesizes training instances for querying to increase learning speed. We generate queries according to the uncertainty principle, but our…
Several recent papers have examined generalization in reinforcement learning (RL), by proposing new environments or ways to add noise to existing environments, then benchmarking algorithms and model architectures on those environments. We discuss subtle conceptual properties of RL benchmarks that are not required in su…
This paper proposes a novel architecture, termed multiscale principle of relevant information (MPRI), to learn discriminative spectral-spatial features for hyperspectral image (HSI) classification. MPRI inherits the merits of the principle of relevant information (PRI) to effectively extract multiscale information embe…
New principle for disentangling latent factors using sparse regularization.
problem Disentangling latent factors from complex data.
method Sparse regularization of latent mechanisms to induce disentanglement.
result Recovery of latent variables up to permutation under certain conditions.
New framework improves model robustness by focusing on stable relations across environments.
problem Standard supervised learning fails under data distribution shift.
method Gradient-based learning framework derived from the principle of independent causal mechanisms (ICM).
result Models generalize well to unseen scenarios, ignoring unstable relations.
This research explores inductive biases for deep learning to improve AI's higher-level cognition.
problem Current AI struggles with flexible out-of-distribution and systematic generalization.
method Examines and proposes new inductive biases for deep learning.
result Identifies specific inductive biases for higher-level sequential processing.
This work uses statistical mechanics to explain AI learning.
problem Understanding the statistical principles behind AI learning.
method Starting from sample concentration behaviors, the study applies statistical mechanics principles to AI and machine learning.
result Exponential families and statistical quantities are key in AI and machine learning.
This research explores principles of Lipschitz continuity in neural networks for robustness and generalization.
problem Ensuring robustness and generalization in neural networks, especially to small input perturbations and out-of-distribution data.
method Two complementary perspectives: internal (training dynamics) and external (frequency signal propagation).
result Advances in understanding the principles of Lipschitz continuity in neural networks.
MEP-Net uses MEP to generate solutions from limited data.
problem Generating solutions to scientific problems with incomplete information.
method Combines MEP with neural networks to learn complex distributions from moment constraints.
result Demonstrates MEP-Net's effectiveness in modeling biochemical reaction networks and generating complex distributions.
Unified geometric principles unify neural network architectures.
problem High-dimensional learning tasks with underlying low-dimensionality and structure.
method Unified geometric principles applied to neural network architectures.
result Unified mathematical framework for neural network architectures.
Machine learning (ML) methods have been developing rapidly, but configuring and selecting proper methods to achieve a desired performance is increasingly difficult and tedious. To address this challenge, automated machine learning (AutoML) has emerged, which aims to generate satisfactory ML configurations for given tas…
Unified approach to causal representation learning using invariance principles.
problem Identifying latent causal variables from high-dimensional observations.
method Guiding identification of causal variables with invariance principles rather than causal hierarchies.
result Unified method that mixes causal and non-causal assumptions improves treatment effect estimation.
New deep network derived from rate reduction principles, explaining features and efficiency.
problem Understanding and optimizing deep learning architectures.
method Gradient ascent scheme for rate reduction leading to multi-layer deep network.
result Explicitly constructed multi-layer network with precise optimization and interpretation.
New method prevents forgetting in learning new tasks.
problem Learning new tasks without forgetting old knowledge.
method Hierarchical information-theoretic optimality principle and Mixture-of-Variational-Experts layer.
result Our method can operate task-agnostically and improve performance in various learning problems.
Maximizes coding rate difference for robust, discriminative features.
problem Learning robust, discriminative features from high-dimensional data.
method Maximal Coding Rate Reduction (MCR^2) principle.
result Significantly more robust to label corruptions in classification.
Proposes a Gaussian process model for constrained dynamics learning.
problem Challenges in identifying constrained dynamics of mechanical systems.
method Combines analytical mechanics with Gaussian process regression.
result Improves data efficiency and constraint integrity in predictions.
This work redefines data-centric AI by unifying categorical and cochain notions.
problem The need to rethink data notions for data-centric AI.
method Proposes unifying principles from categorical and cochain notions of data.
result Unified definition of data impacts machine learning development, implementation, and utilization.
Deep neural networks' infinite-width behavior approximated by Gaussian models.
problem Understanding the behavior of deep neural networks in the limit of infinite width.
method Using the Lindeberg exchange principle to approximate weights by Gaussian random variables.
result Quantitative bounds on the 2-Wasserstein distance between deep neural networks and Gaussian limits.
Framework selects optimal historical data windows for non-stationary learning.
problem Learning in environments where conditions change over time.
method Stability principle applied to select look-back windows.
result Regret bounds are minimax optimal for strongly convex or Lipschitz population losses.
A new principle for extrapolating regression outside training data.
problem Regression extrapolation when predictions are outside training data range.
method Data-adaptive marginal transformation and simple relationship assumption.
result Progression method offers guarantees on approximation error beyond training data range.
Quantum machine learning uses quantum cross entropy to minimize loss, but measurement loss affects this process.
problem Quantum machine learning's loss minimization through cross entropy is affected by measurement outcomes.
method Defined quantum cross entropy, proved its lower bounds, and investigated its relation to quantum fidelity and likelihood.
result Quantum cross entropy is lower-bounded by negative log-likelihood when derived from quantum data, but measurement outcomes can cause loss.
In this paper, we are proposing a unified and principled method for both the querying and training processes in deep batch active learning. We are providing theoretical insights from the intuition of modeling the interactive procedure in active learning as distribution matching, by adopting the Wasserstein distance. As…