This paper surveys informed machine learning, integrating prior knowledge into ML.
problem Machine learning's limitations with insufficient data.
method Taxonomy and survey of informed machine learning approaches.
result A taxonomy classifies informed machine learning approaches based on knowledge source, representation, and integration.
Machine learning approximates phase transitions using Fisher information.
problem Understanding phase transitions from data using machine learning.
method Information geometry and Fisher information.
result Machine learning indicators approximate the square root of Fisher information.
Extends machine learning model training to include validation data.
problem Machine learning validation data usage in training.
method Extends predictive information bottleneck to include validation data.
result Validation data improves model training efficiency.
This paper reviews information theory in open-world machine learning.
problem Lack of a unified theoretical foundation for open-world machine learning.
method Synthesis of information theoretic approaches.
result Established a pathway toward provable and trustworthy open world intelligence.
New model quantifies how much machine learning models can reveal about individual data usage.
problem Measuring and reducing the leakage of membership information from machine learning models.
method Using information theory, conditional mutual information leakage, and Kullback-Leibler divergence to quantify and bound the leakage.
result The amount of membership information leakage is reduced by adding Gaussian (ε,δ)-differentially-private additive noises. New bound on machine learning model performance using Jensen-Shannon information.
problem Understanding the performance of machine learning models.
method Proposes a new information-theoretic bound on generalization error.
result Shows that the new bound can be tighter than mutual information-based bounds under certain conditions.
Survey of integrating physics knowledge into machine learning models.
problem Mitigating data shortage and ensuring physical plausibility.
method Combining physics knowledge with machine learning models.
result Summarizes recent works in physics-informed machine learning.
MetaPhysiCa tackles robust physics-informed machine learning for OOD tasks.
problem Designing robust PIML methods for OOD forecasting tasks in physics.
method Meta-learning procedure for causal structure discovery including invariant risk minimization.
result Significantly outperforms existing PIML and deep learning methods in OOD tasks.
Physics-informed machine learning models improve biomolecular system simulations.
problem Modeling unresolved interactions beyond classical force fields.
method Physics-informed neural networks and operator learning.
result Accurate, mechanistic, generalizable models for long-timescale kinetics.
Framework for understanding overfitting and underfitting using information theory.
problem Understanding and preventing overfitting and underfitting in machine learning.
method Information-theoretic framework measuring algorithm capacity and dataset information transfer.
result Upper-bounding algorithm capacity and establishing its relationship to machine learning quantities.
Improves machine learning models by incorporating physical laws into feature maps.
problem Lack of model interpretability in classical machine learning approaches.
method Physics-informed feature maps constructed from physical laws and dimensional analysis.
result Enhanced model interpretability and potential discovery of new physical equations.
Unified framework for removing unwanted information from machine learning models.
problem Removing undesirable features or data points from machine learning models while preserving utility.
method Information-theoretic regularization approach for data point and feature unlearning.
result Unified mathematical framework with provable guarantees for both data point and feature unlearning.
A model for human-machine decision-making with private info and opacity.
problem Optimizing decisions in a human-machine system with private info and opacity.
method Formulated as a two-player learning problem, proved lower and upper bounds on optimality.
result Simple coordination strategy is nearly minimax optimal, efficient learning possible under certain assumptions.
New insights into encoder-decoder structures using information measures.
problem Understanding the role of encoder-decoder design in machine learning.
method Using information sufficiency and mutual information loss concepts.
result Characterizes the expressiveness loss in encoder-decoder designs.
New algorithm uses machine learning to predict high-frequency trading returns.
problem Improving prediction accuracy in high-frequency trading.
method Iterative optimization and activation functions in deep learning, combined with VPIN, GARCH, and SVM. result The model significantly improved prediction of market liquidity and trading returns.
New method detects information leakage using approximate Bayes predictor.
problem Unintentional exposure of sensitive information via observable data.
method Statistical learning theory and information theory framework, approximating Bayes predictor's log-loss and accuracy.
result MI can be accurately estimated to detect ILs, outperforming state-of-the-art baselines.
Physics-informed kernel learning integrates physical priors into machine learning models.
problem Tackles the integration of physical laws into machine learning models for improved accuracy and efficiency.
method Uses Fourier methods to approximate the kernel and minimizes a physics-informed risk function.
result Demonstrates PIKL outperforms physics-informed neural networks and traditional PDE solvers in various scenarios.
A new framework uses uncertainty to learn from raw data without explicit models.
problem Limitations of traditional machine learning models and lack of interpretability.
method Introduces a model-free framework using surprisal (information theoretic uncertainty) to analyze and infer from raw data.
result Achieves at or near state-of-the-art performance across various machine learning tasks.
QFDA combines machine learning and information theory for image classification.
problem Lack of literature on combining machine learning and information theory.
method Quantized Fisher Discriminant Analysis (QFDA) using a cost function for rate-distortion optimization.
result QFDA achieves at least as good classification accuracy as FDA on quantized images.
Unified notation simplifies information-theoretic concepts in machine learning.
problem Opaque notation for information-theoretic quantities in machine learning.
method Proposed a practical and unified notation for information-theoretic quantities.
result Unified notation facilitates new intuitions and rederivations in machine learning.
This chapter reviews classic regression methods and their evolution to physics-informed approaches.
problem Finding relationships between variables using regression.
method Introduces traditional and physics-informed regression methods, linking them to computational science.
result Regression methods have evolved from purely statistical to incorporating physical knowledge.
Paper uses machine learning in EM framework for better nowcasting.
problem Improving risk estimation with incomplete information.
method Expectation-maximisation framework with machine learning for occurrence and reporting modeling.
result XGBoost-based approach outperforms existing methods in nowcasting.
New graph embedding method uses domain-specific knowledge.
problem Lack of semantic information in existing graph embedding approaches.
method Domain-aware biased random walks to incorporate semantic information.
result Embeddings achieve equal or greater accuracy than domain-independent methods.
The paper introduces submodular information measures for machine learning applications.
problem Generalizing information-theoretic measures to non-random variables.
method Developing combinatorial information measures based on submodular functions.
result Submodular mutual information is submodular in one argument for certain submodular functions.
This paper analyzes machine learning workflows in climate modeling.
problem Challenges in integrating machine learning with climate modeling.
method Analysis of case studies focusing on design patterns and workflow structure.
result Synthesis of workflow design patterns across diverse projects in ML-enabled climate modeling.
Machine Learning benefits from prior information and computational power for better performance and understanding.
problem Improper use of Machine Learning methods leads to lack of understanding and performance issues.
method Employing prior information and computational power to solve learning problems, emphasizing interpretability and performance.
result Combining prior information and computational power can lead to better understanding and performance in Machine Learning.
Unified treatment of PAC-Bayesian and information-theoretic generalization bounds.
problem Generalization capabilities of machine learning algorithms.
method PAC-Bayesian and information-theoretic perspectives.
result Unified treatment and modular structure of proofs.
Visualizes information flow in ML systems for better understanding and analysis.
problem Understanding the flow of information in complex ML systems.
method Proposes a visual approach using Sankey Diagrams to analyze flow of information.
result Demonstrates the effectiveness of the proposed technique in diagnosing model performance.
MPWTSVM improves multi-view learning by reducing redundancy and enhancing accuracy.
problem Improving multi-view learning models for better accuracy and efficiency.
method Proposes MPWTSVM, which combines WLTSVM's strengths with multi-view learning principles.
result Demonstrates better accuracy and efficiency compared to existing multi-view classification models.
We introduce DQFIM to quantify and improve generalization of quantum machine learning models.
problem Understanding and improving generalization of quantum machine learning models.
method Data quantum Fisher information metric (DQFIM) to quantify circuit parameters and training data.
result Improves generalization by breaking symmetries of training data and using a low number of training states.
Study examines explainable machine learning for monotonic models, finding Integrated gradients better for strong monotonicity.
problem Applying explainable machine learning to science-informed models.
method Proposed axioms for monotonicity, tested Shapley value and Integrated gradients methods.
result Integrated gradients provides better explanations for strong monotonicity.
Interpretable machine learning has gained much attention recently. Briefness and comprehensiveness are necessary in order to provide a large amount of information concisely when explaining a black-box decision system. However, existing interpretable machine learning methods fail to consider briefness and comprehensiven…
Quantum TNCS uses machine learning to efficiently transmit data.
problem Efficient quantum communication of large datasets.
method Combining compressed sensing, tensor networks, and machine learning.
result High efficiency and accuracy in transmitting information.
New measures for causal entropy and information gain studied.
problem Quantifying causal relationships in machine learning.
method Formal study of causal entropy and information gain.
result Established fundamental properties and relationships.
New bounds on machine learning data leakage identified.
problem Machine Learning models can leak sensitive information.
method Formalized attack setups, derived universal bounds, studied mutual information.
result Connected attack success rate to generalization gap and mutual information.
PIML uses physics equations in machine learning for better forecasting.
problem Forecasting time series data with physical constraints.
method Physics-informed neural networks (PINNs) and kernel methods.
result PIML improves forecasting accuracy with physical constraints.
Distillation (Hinton et al., 2015) and privileged information (Vapnik & Izmailov, 2015) are two techniques that enable machines to learn from other machines. This paper unifies these two techniques into generalized distillation, a framework to learn from multiple machines and data representations. We provide theoretica…
The relevance of data quantifies learning efficiency.
problem Understanding the statistical nature of high-dimensional, sparse data.
method Defining relevance as information content, and using it to define ideal limits of samples and learning machines.
result Maximally informative samples and optimal learning machines exhibit critical features like power-law frequency distributions and anomalously large susceptibility.
KOC+ uses privileged information to improve one-class classification performance.
problem Outlier detection and novelty detection using kernel methods.
method Kernel ridge regression with correction function for privileged information.
result KOC+ achieves better generalization performance compared to traditional methods.
A new method reduces energy consumption in machine learning by using multiple, less costly data sources.
problem High computational and energy costs in machine learning model training.
method Augmented Gaussian Process (AGP-MISO) with multi-source optimization.
result The AGP-MISO method reduces computational time and energy consumption compared to traditional approaches.
The paper proposes a bio-inspired framework for better compression and adversarial robustness in machine learning models.
problem Machine learning models are vulnerable to adversarial examples.
method The paper introduces a bio-inspired classification framework that conditions model inference on label hypothesis and uses an information bottleneck regularizer.
result The framework enables better compression and adversarial robustness without loss of natural accuracy.
PICN learns physical fields from shallow neural networks, improving AI in multi-physical systems.
problem Challenges in modeling and forecasting multi-physical systems due to data scarcity and noise.
method Physics-informed convolutional network (PICN) combining CNN and physical laws, using deconvolution and convolution layers.
result PICN effectively solves and estimates nonlinear physical operator equations and recovers physical information from noisy observations.
This work uses statistical mechanics to explain AI learning.
problem Understanding the statistical principles behind AI learning.
method Starting from sample concentration behaviors, the study applies statistical mechanics principles to AI and machine learning.
result Exponential families and statistical quantities are key in AI and machine learning.
p3VAE combines physics and machine learning for robust data representations.
problem Improving machine learning models' robustness to environmental factors of variation.
method Physics-informed variational autoencoder integrating physical knowledge with neural networks.
result p3VAE outperforms competing models in extrapolation and interpretability. The paper analyzes how incentives and information affect ROC curves in medical diagnosis.
problem Understanding how incentives and information asymmetry influence ROC curves in medical diagnosis.
method Theoretical analysis and illustration with a real-world dataset.
result Different models of incentives and information asymmetry affect the relationship between human decisions and ROC curves.
We are working to develop automated intelligent agents, which can act and react as learning machines with minimal human intervention. To accomplish this, an intelligent agent is viewed as a question-asking machine, which is designed by coupling the processes of inference and inquiry to form a model-based learning unit.…
New framework ties learning algorithms to data recognition using CMI.
problem Understanding how well machine learning models generalize from training data.
method Information-theoretic framework using Conditional Mutual Information (CMI).
result Bounds on CMI imply various forms of generalization.
Introduces a framework using information theory for understanding machine learning.
problem Understanding the effectiveness and design of modern machine learning architectures.
method An information-theoretic approach to learning, focusing on model complexity and architecture.
result Successful architectures have a broad complexity range, enabling learning in over-parameterized model classes.