Gradient descent in logistic regression converges to the maximum margin predictor.
problem Convergence and risk of logistic regression parameters.
method Gradient descent applied to logistic regression.
result Gradient descent iterates converge to the maximum margin predictor at a rate of O(lnlnt/lnt). New method calculates partial information for Gaussian systems based on dependency constraints.
problem Quantifying information sharing in multivariate Gaussian systems.
method Constructing maximum entropy models based on dependency constraints and deriving closed-form solutions.
result Closed-form solutions for Gaussian systems show differences in redundancy and synergy estimates compared to existing methods.
A new method for unsupervised domain adaptation using Gaussian processes.
problem Reducing target domain error by aligning input and output distributions.
method Max-margin Gaussian process approach to achieve hypothesis consistency.
result Our method effectively minimizes maximum discrepancy and maximizes margins.
The study identifies factors predicting stock returns and maximum drawdown using various models.
problem Predicting stock returns and maximum drawdown in the US equity market.
method Supervised learning with multiple models (OLS, penalized linear regressions, tree-based models, neural networks) over 49 years of data.
result Non-linear models outperformed linear models in predicting stock returns and maximum drawdown, especially during calm periods.
New bounds show linear predictors rarely overfit with certain optimization methods.
problem Bounding test error for linear predictors with stochastic optimization methods.
method Coupling argument for fixed point methods like stochastic and batch mirror descent.
result Locally-adapted rates that depend on predictor properties, not global problem structure.
It has been argued that in supervised classification tasks, in practice it may be more sensible to perform model selection with respect to some more focused model selection score, like the supervised (conditional) marginal likelihood, than with respect to the standard marginal likelihood criterion. However, for most Ba…
A new classifier updates sequentially using maximum margin principles.
problem Sequential data collection and partial labeling.
method Maximum margin classifier with Maximum Entropy Discrimination principle, kernel representation, and regularization.
result Improved performance compared to non-sequential classifiers.
Study connects spectral clustering to maximum margin and level set estimation.
problem Connecting spectral clustering to maximum margin and level set estimation.
method Obtained bounds on eigenvectors of graph Laplacian matrices in terms of cluster separation and connectivity. Showed sensitivity mitigation by removing outliers and estimating level sets.
result Spectral clustering converges to maximum margin clustering as scaling parameter approaches zero.
New research shows the maximum ℓ1-margin classifier doesn't adapt to sparse ground truths.
problem Understanding the limitations of the maximum ℓ1-margin classifier in high-dimensional settings.
method Analyzing convergence and prediction error rates of the maximum ℓ1-margin classifier.
result Proves tight upper and lower bounds for prediction error, showing benign overfitting.
New probabilistic complexity measures for linear and kernel methods.
problem Limitations of linear and kernel methods in machine learning.
method Introducing approximate notions of dimensional and margin complexity.
result Approximate complexity measures are both sufficient and necessary for learning.
Gradient methods avoid overfitting on separable data.
problem Overfitting on separable data with gradient methods.
method Gradient methods (flow, descent, SGD) on separable data.
result Gradient methods asymptotically do not overfit on separable data.
The problem of determining the joint probability distributions for correlated random variables with pre-specified marginals is considered. When the joint distribution satisfying all the required conditions is not unique, the "most unbiased" choice corresponds to the distribution of maximum entropy. The calculation of t…
Study develops large margin machine learning models for predicting host-pathogen protein interactions.
problem Identifying host-pathogen interactions to develop new drugs for infectious diseases.
method Large margin machine learning models, specifically SVM with weighted negative sampling and distance-based weight assignment.
result Proposed and validated a new method for predicting host-pathogen protein interactions.
The paper analyzes the maximum margin algorithm's performance on noisy data.
problem Analyzing the performance of maximum margin algorithm on noisy data.
method Finite-sample analysis of maximum margin algorithm applied to noisy data.
result The maximum margin algorithm can achieve nearly optimal population risk with sufficient over-parameterization.
We give polynomial-time algorithms for the exact computation of lowest-energy (ground) states, worst margin violators, log partition functions, and marginal edge probabilities in certain binary undirected graphical models. Our approach provides an interesting alternative to the well-known graph cut paradigm in that it …
Group model selection is the problem of determining a small subset of groups of predictors (e.g., the expression data of genes) that are responsible for majority of the variation in a response variable (e.g., the malignancy of a tumor). This paper focuses on group model selection in high-dimensional linear models, in w…
We obtain bounds on the distribution of the maximum of a martingale with fixed marginals at finitely many intermediate times. The bounds are sharp and attained by a solution to n-marginal Skorokhod embedding problem in Obłój and Spoida [An iterated Azéma-Yor type embedding for finitely many marginals (2013) Preprint]…
New bounds explain deterministic non-smooth deep nets without large Lipschitz constants.
problem Challenges in explaining generalization of deterministic non-smooth deep nets.
method De-randomized PAC-Bayes margin bounds for deterministic non-convex and non-smooth predictors.
result New bounds avoid large Lipschitz constants, providing generalization guarantees.
Bayesian models use hyperparameters to indirectly assign priors, and this work shows how these priors can be derived from maximum entropy principles.
problem Understanding the assumptions and dependencies in Bayesian hierarchical models.
method Demonstrates how canonical distributions and maximum entropy principles can be used to derive marginal priors in hierarchical models.
result Marginal priors in hierarchical models derived from maximum entropy principles have different constraints compared to the original priors.
Study shows how over-parameterized classifiers can still perform well on noisy data.
problem Understanding how maximum margin classifiers perform in over-parameterized settings with noisy data.
method Analyzes maximum margin classifiers on sub-Gaussian mixtures, providing risk bounds.
result Characterizes conditions for 'benign overfitting' in linear classification problems.
A method for selecting disease-related genes from multiple experiments.
problem Determining which genes affect a disease from various experiments.
method Formulating a general approach to select informative genes from mixed data types, combining marginal likelihoods and proposing a pseudolikelihood information criterion.
result The method improves gene selection from multiple experiments compared to using a single experiment.
Developed R package for creating nomograms for any ML algorithms.
problem Creating nomograms for any machine learning algorithms.
method Formulated a function to transform ML prediction models into nomograms, requiring specific datasets.
result Created 5 types of nomograms for various ML algorithms and predictor types.
A framework estimates categorical distributions under constraints, ensuring generality and uniqueness.
problem Estimating categorical distributions summarizing sample data under marginal constraints.
method Theoretical framework + Iterative Proportional Fitting (IPF) to estimate the distribution.
result A unique categorical distribution of Maximum Entropy under marginal constraints exists and is estimated.
Gradient descent implicitly follows regularization for general losses.
problem The implicit bias of gradient descent methods in machine learning.
method Empirical risk minimization over linear predictors with arbitrary convex, strictly decreasing losses.
result Gradient descent and regularization paths converge to the same direction for non-attained risks.
This work analyzes the maximum-margin bias in quasi-homogeneous neural networks.
problem Analyzing the maximum-margin bias in quasi-homogeneous neural networks.
method Geometric analysis of gradient dynamics for quasi-homogeneous models.
result Gradient flow implicitly favors a subset of parameters, leading to asymmetric norm minimization.
A new method improves maximum margin criterion for better pattern analysis.
problem Handling high dimensionality and large data in pattern analysis.
method Introducing an improved maximum margin criterion (MMC) and its variants.
result Experimental results show the MMC methods are effective in complex scenarios.
Gradient descent on separable data converges to SVM's max-margin solution.
problem Gradient descent on logistic regression problems with linearly separable data.
method Examination of gradient descent on logistic regression with homogeneous predictors.
result Gradient descent converges to the max-margin solution of SVM.
Gradient descent on logistic loss converges to the maximum-margin separator for separable data.
problem Understanding the convergence of gradient descent on separable datasets with specific loss functions.
method Analysis of gradient descent on linear models with super-polynomially tailed losses.
result For separable datasets, gradient descent converges to the maximum-margin separator for losses with super-polynomial tails, but not for heavier tails.
New margin-based learning guarantees improve generalization bounds.
problem Improving generalization bounds for machine learning models.
method Relative deviation margin bounds using empirical margin loss and Rademacher complexity.
result Distribution-dependent generalization bounds for unbounded loss functions.
Graphical models with bi-directed edges (<->) represent marginal independence: the absence of an edge between two vertices indicates that the corresponding variables are marginally independent. In this paper, we consider maximum likelihood estimation in the case of continuous variables with a Gaussian joint distributio…
Method screens weakly associated predictors in high-dimensional data.
problem Identifying weakly associated predictors in ultrahigh-dimensional data.
method Covariance-insured screening methodology.
result Validates the method through simulations and real data studies.
Supervised topic models utilize document's side information for discovering predictive low dimensional representations of documents. Existing models apply the likelihood-based estimation. In this paper, we present a general framework of max-margin supervised topic models for both continuous and categorical response var…
Causal invariance can improve finite-sample domain adaptation, but only when the target risk margins are large.
problem Finite-sample domain adaptation
method Linear regression with causal knowledge
result Adaptive aggregation can match best candidate predictor while avoiding negative transfer
The paper presents a method to estimate joint interventional distributions from marginal interventional data.
problem Estimating joint interventional distributions from marginal interventional data.
method The paper extends the Causal Maximum Entropy method to use interventional data and employs Lagrange duality to prove the solution lies in the exponential family.
result The method allows for causal feature selection and inference of joint interventional distributions.
Study on merging predictors in causal and anticausal directions using CMAXENT.
problem Comparing merging predictors in causal and anticausal directions.
method Using CMAXENT as inductive bias, study differences in merging predictors.
result CMAXENT solution reduces to logistic regression in causal direction and LDA in anticausal direction.
CAOS aggregates multiple one-shot predictors for efficient uncertainty quantification.
problem Lack of principled uncertainty quantification in one-shot prediction.
method CAOS, a conformal framework that aggregates multiple one-shot predictors and uses a leave-one-out calibration scheme.
result CAOS produces smaller prediction sets with reliable coverage compared to split conformal baselines.
New algorithm improves latent variable model estimation.
problem Estimating parameters in latent variable models.
method Jarzynski-adjusted Langevin algorithm (JALA) for SMC methods.
result JALA-EM provides maximum marginal likelihood estimate.
A fast method for training linear classifiers maximizes margins.
problem Training linear classifiers with maximum margins.
method Momentum-based gradient method derived from convex dual with Nesterov acceleration.
result Exponentially faster convergence rate compared to standard methods.
Mirror flow optimizes separable data problems, converging to a maximum margin classifier.
problem Optimizing classification problems with separable data using mirror flow.
method Examine mirror flow on linearly separable classification problems, focusing on the horizon function of the mirror potential.
result Mirror flow converges to a maximum margin classifier for separable data under certain conditions.
Optimizes risk measures given known marginal distributions of two unknown factors.
problem Determining an upper bound for spectral risk measures with unknown joint distribution.
method Introduces Maximum Spectral Measure (MSP) as a worst-case risk measure, formulated as an optimization problem with a more general objective function.
result Characterizes the continuity properties of the optimal value function and optimal solution set with respect to marginal distributions.
Paper introduces new importance metrics for machine learning models, linking them to CATE.
problem Interpreting black-box models' importance metrics due to data dependence and non-parametric nature.
method Introduces MVIM and CVIM, proposing permutation-based estimation and bias-variance decomposition.
result MVIM and CVIM have a quadratic relationship with CATE, addressing bias in correlated predictors.
We consider two connected aspects of maximum likelihood estimation of the parameter for high-dimensional discrete graphical models: the existence of the maximum likelihood estimate (mle) and its computation. When the data is sparse, there are many zeros in the contingency table and the maximum likelihood estimate of th…
We consider the problem of the optimal trading strategy in the presence of linear costs, and with a strict cap on the allowed position in the market. Using Bellman's backward recursion method, we show that the optimal strategy is to switch between the maximum allowed long position and the maximum allowed short position…
New method resolves nonidentifiability in mixture models.
problem Nonidentifiability in marginal models of mixture models.
method Introducing an effective temperature to generalize the marginal likelihood.
result Maximization of the generalized likelihood leads to unique results.
We consider the problem of learning Bayesian network classifiers that maximize the marginover a set of classification variables. We find that this problem is harder for Bayesian networks than for undirected graphical models like maximum margin Markov networks. The main difficulty is that the parameters in a Bayesian ne…
Paper introduces a new margin bound for neural networks scaling with spectral complexity.
problem Improving generalization bounds for neural networks.
method Spectral complexity is defined as the product of the spectral norms of weight matrices, scaled by a correction factor.
result Empirical investigation shows correlation between bound, complexity, and excess risk for SGD-trained AlexNet.
We introduce a new approach to variable selection, called Predictive Correlation Screening, for predictor design. Predictive Correlation Screening (PCS) implements false positive control on the selected variables, is well suited to small sample sizes, and is scalable to high dimensions. We establish asymptotic bounds f…
Unified framework for interpreting complex regression models with many predictors.
problem Interpreting nonparametric regression models with many predictors.
method Derivative-based approach for existing tools like partial-dependence plots.
result New technique called accumulated total derivative effects plot for complex models.