Study on how optimal representations emerge during deep learning training, focusing on the role of implicit regularization.
problem Understanding how optimal representations for tasks are learned during training.
method Investigates the role of implicit regularization in learning minimal sufficient representations, analyzing changes in representation content during training.
result Semantically meaningful but ultimately irrelevant information is encoded in early transient dynamics of training, which is later discarded.
FSRL balances fairness and sufficiency in learning representations.
problem Ensuring unbiased predictions and decisions in machine learning.
method A convex combination of sufficiency and fairness objectives using distance covariance.
result FSRL achieves a superior trade-off between fairness and accuracy.
We investigate minimal surfaces passing a given curve in R3. Using the Frenet frame of a given curve and isothermal parameter, we derive the necessary and sufficient condition for minimal surface. Also we derive the parametric representation of two minimal surface families passing a circle and a helix as examples.
One of the most fundamental questions one can ask about a pair of random variables X and Y is the value of their mutual information. Unfortunately, this task is often stymied by the extremely large dimension of the variables. We might hope to replace each variable by a lower-dimensional representation that preserves th…
Study feature representations induced by dependence between variables.
problem Learning feature representations from dependent random variables.
method Characterized sufficient and necessary conditions for dependence-induced representations, and provided a family of loss functions.
result Features learned from the family of loss functions can be expressed as the composition of a loss-dependent function and the maximal correlation function.
We consider a general regularised interpolation problem for learning a parameter vector from data. The well known representer theorem says that under certain conditions on the regulariser there exists a solution in the linear span of the data points. This is at the core of kernel methods in machine learning as it makes…
New statistical theory explains contrastive learning effectiveness.
problem Understanding why contrastive learning works well for representation extraction.
method Developed a new theoretical framework based on approximate sufficient statistics.
result Near-sufficient encoders derived from contrastive learning can be adapted for downstream tasks.
Proposes a deep learning method for effective data representation.
problem Constructing effective data representations for prediction.
method A deep dimension reduction approach to learning representations with sufficiency, low dimensionality, and disentanglement.
result The proposed deep nonparametric representation is consistent and performs better than existing methods.
Given two univalent harmonic mappings f1 and f2 on D, which lift to minimal surfaces via the Weierstrass-Enneper representation theorem, we give necessary and sufficient conditions for f3=(1−s)f1+sf2 to lift to a minimal surface for s∈[0,1]. We then construct such mappings from Enneper's surfa…
A method identifies domain-general features using causal graph constraints and regularization.
problem Identifying domain-general features without prior knowledge of spurious features.
method Proposes a novel regularization framework based on causal graph constraints.
result Demonstrates effectiveness in both synthetic and real-world data, outperforming state-of-the-art methods.
Study minimizes risk in MDPs with spectral measures.
problem Minimizing risk in MDPs with spectral measures.
method Splitting into inner and outer minimization problems; solving inner as MDP; proving existence for outer.
result Existence and solution methods for the outer minimization problem.
Paper solves a conjecture about minimal surfaces using sphere intersections and Weierstrass data.
problem Solving the Fraser-Li conjecture for minimal surfaces.
method Using the Weierstrass representation formula and sphere intersections.
result The conjecture can be translated into problems about the Gauss map.
This article studies quadratic semimartingale BSDEs arising in power utility maximization when the market price of risk is of BMO type. In a Brownian setting we provide a necessary and sufficient condition for the existence of a solution but show that uniqueness fails to hold in the sense that there exists a continuum …
FedLog reduces communication in federated learning by sharing data summaries.
problem Significant communication overhead in federated learning with large model parameters.
method Shares minimal sufficient statistics via Bayesian inference and differential privacy.
result High learning accuracy with low communication overhead.
Boosting improves data fitting while maintaining fairness guarantees.
problem Ensuring fairness in data preprocessing.
method Boosting algorithm to learn sufficient statistics of exponential families.
result The learned distribution maintains fairness guarantees while fitting the data better.
Paper develops a theory explaining contrastive pre-training for multimodal AI.
problem Limited theoretical understanding of contrastive pre-training for multi-modal AI.
method Introduces approximate sufficient statistics and Joint Generative Hierarchical Model.
result Near-minimizers of contrastive loss are approximately sufficient, enabling diverse downstream tasks.
GALA framework learns invariant graph representations via environment augmentation with minimal assumptions.
problem Learning invariant graph representations from different environments without additional assumptions.
method Developed GALA framework with minimal assumptions of variation sufficiency and consistency. Uses an assistant model to differentiate graph environment changes.
result Extracting maximally invariant subgraphs to proxy predictions identifies underlying invariant subgraphs for successful out-of-distribution generalization.
The Information Bottleneck (IB) is a conceptual method for extracting the most compact, yet informative, representation of a set of variables, with respect to the target. It generalizes the notion of minimal sufficient statistics from classical parametric statistics to a broader information-theoretic sense. The IB curv…
In this study, a novel feature coding method that exploits invariance for transformations represented by a finite group of orthogonal matrices is proposed. We prove that the group-invariant feature vector contains sufficient discriminative information when learning a linear classifier using convex loss minimization. Ba…
SEEK algorithm selects minimal state in reinforcement learning for better policy learning.
problem Challenges in obtaining a state representation that is parsimonious and satisfies the Markov property.
method SEEK algorithm estimates the minimal sufficient state in reinforcement learning.
result The SEEK algorithm achieves selection consistency in large samples.
MSRL learns a representation maximizing mutual info with response variables.
problem Learning sufficient representations for complex, multi-dimensional data.
method Variational mutual information, deep neural networks, generalized Dudley's inequality.
result MSRL achieves consistent and accurate representation learning.
This paper considers compressed sensing and affine rank minimization in both noiseless and noisy cases and establishes sharp restricted isometry conditions for sparse signal and low-rank matrix recovery. The analysis relies on a key technical tool which represents points in a polytope by convex combinations of sparse v…
NICE learns a representation to avoid bad controls in causal inference.
problem Avoiding bad controls in causal inference from observational data.
method Uses invariant risk minimization (IRM) to learn a representation of covariates that avoids bad controls.
result NICE outperforms adjusting for all covariates in cases with unknown collider variables and bad controls.
Study shows how information loss and operation loss are related in feature representations.
problem Understanding the relationship between information loss and operation loss in feature representations.
method Analyzes the interplay between weak information loss and operation loss in continuous representations.
result Specific forms of vanishing information loss imply vanishing MPE loss in classification.
Paper recovers latent causal structure and linear transformation from indirect observations.
problem Recovering latent causal structure and linear transformation from indirect observations.
method Established sufficient conditions for DAG recovery, leveraged score function properties, and used soft/hard interventions.
result Perfect recovery of latent DAG structure and linear transformation up to scaling using soft interventions, hard interventions with additional hypothesis testing.
We address some theoretical guarantees for Schatten-p quasi-norm minimization (p∈(0,1]) in recovering low-rank matrices from compressed linear measurements. Firstly, using null space properties of the measurement operator, we provide a sufficient condition for exact recovery of low-rank matrices. This condition…
We propose approaches based on deep learning to localize objects in images when only a small training dataset is available and the images have low quality. That applies to many problems in medical image processing, and in particular to the analysis of fluoroscopic (low-dose) X-ray images, where the images have low cont…
Unified framework for fair representation learning in machine learning.
problem Ensuring fairness in machine learning models, especially when biased data representations lead to unfair predictions.
method Integrates nonlinear sufficient dimension reduction with deep learning to construct fair and informative representations, introducing a penalty term to enforce conditional independence between sensitive attributes and learned representations.
result Achieves a superior balance between fairness and utility, significantly outperforming state-of-the-art baselines on various data structures.
Study on stability of free boundary Willmore problem using new gradient inequality.
problem Stability of free boundary Willmore problem.
method New Łojasiewicz-Simon gradient inequality for functionals on infinite dimensional manifolds.
result Existence and convergence of solutions for the free boundary Willmore flow.
Enhances source domain knowledge with target data for transfer learning.
problem Limited data in target domains and rigid model assumptions in transfer learning.
method Transfer learning through Enhanced Sufficient Representation (TESR).
result TESR enhances source domain knowledge with target data, improving transfer learning performance.
We introduce Minimal Achievable Sufficient Statistic (MASS) Learning, a training method for machine learning models that attempts to produce minimal sufficient statistics with respect to a class of functions (e.g. deep networks) being optimized over. In deriving MASS Learning, we also introduce Conserved Differential I…
In this paper we study a continuous-time stochastic linear quadratic control problem arising from mathematical finance. We model the asset dynamics with random market coefficients and portfolio strategies with convex constraints. Following the convex duality approach, we show that the necessary and sufficient optimalit…
The study finds conditions for area-minimizing cones over submanifolds.
problem Conditions for area-minimizing cones over submanifolds.
method General configuration results for area-minimizing cones.
result Cone over the minimal product of submanifolds and spheres are area-minimizing.
Analyzes empirical risk minimization in finance, showing effectiveness and generalization issues.
problem Analyzing empirical risk minimization in finance for optimal hedging and investment decisions.
method Classical statistical machine learning techniques and non-asymptotic estimates based on Rademacher complexity.
result Over-training leads to anticipative decisions, but non-asymptotic estimates show convergence for large training sets.
A new criterion for deep active learning selects minimal labeled data points.
problem Efficiently select minimal labeled data points for deep neural networks.
method Diffuses label information over a graph of data representations to switch between exploration and refinement.
result The diffusion-based criterion outperforms existing methods in deep active learning.
ETC learns minimal representations for reinforcement learning in POMDPs.
problem Sample complexity challenges in reinforcement learning for POMDPs.
method ETC learns low-dimensional features and embeddings at two levels, optimizing policy.
result ETC achieves polynomial sample complexity for POMDPs with low-rank transition kernels.
Low-rank matrix recovery has found many applications in science and engineering such as machine learning, signal processing, collaborative filtering, system identification, and Euclidean embedding. But the low-rank matrix recovery problem is an NP hard problem and thus challenging. A commonly used heuristic approach is…
New method uses predictions to infer causal effects without labeled data.
problem Data labeling costs limit causal inference experiments.
method Prediction-Powered Causal Inferences (PPCI) using conditional calibration and transfer constraints.
result Valid causal inference achieved on experiments with no human annotations.
We apply a quadratic hedging scheme developed by Foellmer, Schweizer, and Sondermann to European contingent products whose underlying asset is modeled using a GARCH process and show that local risk-minimizing strategies with respect to the physical measure do exist, even though an associated minimal martingale measure …
TCRI improves domain generalization by enforcing conditional independence constraints.
problem Limitations of existing domain generalization methods due to incomplete constraints.
method TCRI implements regularizers motivated by conditional independence constraints.
result TCRI achieves cross-domain stability and outperforms baselines in worst-domain accuracy.
This paper addresses law invariant coherent risk measures and their Kusuoka representations. By elaborating the existence of a minimal representation we show that every Kusuoka representation can be reduced to its minimal representation. Uniqueness -- in a sense specified in the paper -- of the risk measure's Kusuoka r…
We address the question of detecting minimal virtual diagrams with respect to the number of virtual crossings. This problem is closely connected to the problem of detecting the minimal number of additional intersection points for a generic immersion of a singular link in R2. We tackle this problem by the so-called…
Let G be a real reductive Lie group and τ:G⟶GL(V) be a real reductive representation of G with (restricted) moment map $m_{\ggo}: V-{0} \longrightarrow \ggo$. In this work, we introduce the notion of "nice space" of a real reductive representation to study the problem of how to determine if a G-…
We give an estimate of the first eigenvalue of the Laplace operator on a complete noncompact stable minimal hypersurface M in a complete simply connected Riemannian manifold with pinched negative sectional curvature. In the same ambient space, we prove that if a complete minimal hypersurface M has sufficiently smal…
A theorem simplifies mass-minimizing flat chains' regularity.
problem Understanding the regularity of mass-minimizing flat chains.
method Simple condition for fundamental regularity principle.
result Fundamental regularity principle holds for mass-minimizing chains.
This paper finds ReLU restores symmetry in SCL under class imbalances.
problem Symmetry break in SCL under class imbalances.
method Analytical proof and experiments with ReLU activation and batch selection.
result ReLU restores symmetry in SCL-learned representations without loss in test accuracy.
Unpaired multi-domain causal representation learning is possible with sufficient conditions.
problem Learning shared causal representation from unpaired data across domains.
method Identify sufficient conditions for joint distribution and shared causal graph recovery.
result Practical method to recover shared latent causal graph from marginal distributions.
P-SE explains model decisions with minimal feature subsets and fast estimators.
problem Explain model decisions in regression and classification.
method Probabilistic Sufficient Explanations (P-SE) with random Forests for conditional probability estimation.
result Consistent and efficient explanations for regression and classification models.