A new learning rule consistently reduces error over data samples.
problem Finding a learning rule that consistently reduces error over all data distributions.
method A deterministic, data-dependent partitioning rule that only partitions cyclic intervals with sufficient empirical diversity of labels.
result The expected error is monotone non-increasing with the sample size under every data distribution.
Learning performance can show non-monotonic behavior. That is, more data does not necessarily lead to better models, even on average. We propose three algorithms that take a supervised learning model and make it perform more monotone. We prove consistency and monotonicity with high probability, and evaluate the algorit…
Monotonic relationship found between in-distribution and out-of-distribution performance.
problem Understanding performance of machine learning models under distribution shifts.
method Analyzing ridge-regularized models and linear inverse problems under covariate shift.
result Monotonic relationship between in-distribution and out-of-distribution performance for certain models.
New findings show privacy affects generalization error in a non-monotonic way.
problem Privacy and robustness in distributed learning.
method Theoretical analysis and matching lower/upper bounds on algorithmic stability.
result Generalization error is non-monotonically affected by privacy, depending on noise level.
We propose an algorithm for a family of optimization problems where the objective can be decomposed as a sum of functions with monotonicity properties. The motivating problem is optimization of hyperparameters of machine learning algorithms, where we argue that the objective, validation error, can be decomposed as mono…
Monotone neural networks can approximate and interpolate functions efficiently.
problem Understanding the efficiency and expressiveness of monotone neural networks.
method Solving the monotone interpolation problem using depth-4 networks and comparing size bounds with arbitrary networks.
result Monotone neural networks can approximate and interpolate functions efficiently, but may require exponential size in high dimensions.
Study tackles criterion collapse in learning criteria, showing conditions for loss minimization.
problem Criterion collapse in optimization, focusing on error probability minimizers.
method Analyzes various learning criteria, including DRO, OCE risks, and non-monotonic criteria.
result Non-monotonic criteria can avoid collapse, while monotonic ones cannot.
Investigates probability of error in structured thresholding bandit problems.
problem Probability of misclassifying arms in structured thresholding bandit problems.
method Analyzes two shape constraints: monotonic increasing and concave sequences of arm means.
result Upper and lower bounds for the probability of error match up to constants in the problem dependent regime.
This paper introduces a novel monotone curve estimation framework based on convex duality.
problem Estimating smooth, continuous, and monotonic curves in data.
method Convex duality and optimal transport theories.
result Established statistical guarantees for monotone curve estimates.
Monotone adversarial corruptions degrade optimal learning algorithms.
problem Optimal learning algorithms' reliance on exchangeability and independence is challenged.
method Introduces a monotone adversarial corruption model where an adversary adds monotone corruptions to a clean dataset.
result Optimal learning algorithms achieve suboptimal expected error on new test points.
This work explains how large neural networks generalize well despite overparameterization.
problem Understanding the generalization behavior of large neural networks.
method Theoretical analysis of approximation and generalization errors in regression and classification tasks.
result Deep overparameterized neural networks are statistically consistent across different tasks when regularization is applied.
Fast algorithm recovers principal eigenvector from noisy matrices.
problem Recovering the first principal eigenvector from noisy positive semidefinite matrices.
method Cone projected power iteration algorithm.
result Achieves polynomial time complexity and small error for certain convex cones.
Monotonic Linear Interpolation property in neural networks persists despite non-convexity.
problem Understanding the geometric properties of neural network loss landscapes.
method Tools from differential geometry to analyze the monotonicity of neural network weights.
result Sufficient conditions for the Monotonic Linear Interpolation property under mean squared error.
We consider the learning algorithms under general source condition with the polynomial decay of the eigenvalues of the integral operator in vector-valued function setting. We discuss the upper convergence rates of Tikhonov regularizer under general source condition corresponding to increasing monotone index function. T…
New method warns of counterfactual non-identifiability in DSCMs.
problem Counterfactual inference from observational data is non-identifiable even without unobserved confounding.
method Prove counterfactual identifiability for monotonic generation mechanisms, provide impossibility result for general mechanisms, propose method for estimating worst-case errors.
result Non-identifiability of counterfactual inference from observational data, even in absence of unobserved confounding.
SurvSurf predicts first hitting times for intermittent events without monotonic violations.
problem Predicting first hitting times for intermittent events with monotonicity guarantees.
method Partially monotonic neural network for sequential events, incorporating unobserved events.
result SurvSurf outperforms existing models in MSE and IBS metrics.
Paper tackles ranking items with a semi-random comparison graph and a monotone adversary.
problem Ranking items based on pairwise comparisons from a semi-random comparison graph with a monotone adversary.
method Developed a weighted maximum likelihood estimator (MLE) and an SDP-based approach to reweight the semi-random graph.
result Achieves near-optimal sample complexity, up to a log^2(n) factor, for identifying the top-K preferred items.
Efficient algorithms find optimal monotone transforms for calibration under strictly convex losses.
problem Calibrating estimations to improve performance with monotone transforms.
method Proposed linear-time and space algorithm for finding optimal monotone transforms for specific loss functions. Also proposed an anytime algorithm with linear space and pseudo-linearithmic time complexity.
result Optimal monotone transforms are unique and can be found efficiently for various strictly convex loss functions.
EPEM efficiently estimates parameters for monotone missing data.
problem Efficiently estimating parameters for monotone missing data.
method Derive exact formulas and propose EPEM algorithm for multiple class, monotone missing datasets.
result EPEM reduces error rates significantly and is faster than other methods.
Training on some out-of-distribution data improves generalization error before it deteriorates.
problem Generalization error improves with some out-of-distribution data but deteriorates with more.
method Synthetic datasets and deep networks on computer vision benchmarks.
result Non-monotonic trend in generalization error with OOD samples.
New method for optimizing risk in financial models using Fourier transforms.
problem Optimizing risk in financial models with multi-period mean-CVaR.
method Strictly monotone 2D integration scheme via Fourier-trained transition kernels.
result Established robust and accurate optimization method for financial models.
Gradient descent dynamics in quadratic regression models are analyzed, revealing five phases: monotonic, catapult, periodic, chaotic, and divergent.
problem Analyzing the dynamics of gradient descent in quadratic regression models.
method Fine-grained bifurcation analysis of gradient descent dynamics using a cubic map parameterized by the step-size.
result Gradient descent dynamics in quadratic regression models exhibit five distinct phases: monotonic, catapult, periodic, chaotic, and divergent.
Most recent results in matrix completion assume that the matrix under consideration is low-rank or that the columns are in a union of low-rank subspaces. In real-world settings, however, the linear structure underlying these models is distorted by a (typically unknown) nonlinear transformation. This paper addresses the…
The number of trees T in the random forest (RF) algorithm for supervised learning has to be set by the user. It is controversial whether T should simply be set to the largest computationally manageable value or whether a smaller T may in some cases be better. While the principle underlying bagging is that "more trees a…
Develops multifactor approximations for SVEs with completely monotone kernels.
problem Approximating SVEs with kernels of completely monotone type.
method Multifactor approximation, Euler discretization, L2-estimation, convergence analysis. result New multifactor Euler scheme reduces computational cost and outperforms SVEs for option pricing.
Deep neural networks can generalize by reducing high-frequency noise over time, not always following a monotonic learning bias.
problem Understanding the learning dynamics and generalization of over-parameterized DNNs.
method Experimental analysis of deep double descent, focusing on the spectral bias of DNNs.
result The high-frequency components of DNNs diminish over training, leading to a second descent in test error.
It is widely believed that the prediction accuracy of decision tree models is invariant under any strictly monotone transformation of the individual predictor variables. However, this statement may be false when predicting new observations with values that were not seen in the training-set and are close to the location…
New adaptive test for NPIV models controls size and has superior power.
problem Testing inequality and equality restrictions in nonparametric IV models.
method Adaptive hypothesis test based on modified leave-one-out sample quadratic distance.
result Adaptive test attains the adaptive minimax rate of testing in L2. Study reveals how neural network smoothness affects their vulnerability to adversarial attacks.
problem Understanding adversarial vulnerability in deep learning networks.
method Analysis of manifold smoothness and generalization capability of deep neural networks trained with local errors.
result High generalization accuracy requires a fast power-law decay of eigen-spectrum of hidden representations.
There is no known efficient method for selecting k Gaussian features from n which achieve the lowest Bayesian classification error. We show an example of how greedy algorithms faced with this task are led to give results that are not optimal. This motivates us to propose a more robust approach. We present a Branch and …
The paper addresses monotonicity in machine learning models for fairness and accountability.
problem Ensuring fairness and accountability in transparent machine learning models.
method Study of three types of monotonicity (individual, weak pairwise, strong pairwise) and propose monotonic groves of neural additive models.
result Monotonic groves of neural additive models maintain transparency, accountability, and fairness.
This work uses ANOVA to understand how different factors contribute to test error in machine learning models.
problem Understanding why overparametrized models generalize well despite potentially fitting noise.
method Analysis of variance (ANOVA) to decompose test error into components of variance.
result The interaction between training samples and initialization can dominate variance, and there are phase transitions in variance behavior.
In this paper we consider the cluster estimation problem under the Stochastic Block Model. We show that the semidefinite programming (SDP) formulation for this problem achieves an error rate that decays exponentially in the signal-to-noise ratio. The error bound implies weak recovery in the sparse graph regime with bou…
Probit Monotone BART estimates binary outcomes using monotonic functions.
problem Estimating conditional mean functions for binary outcomes with monotonicity constraints.
method Proposes a new BART variant that incorporates monotonicity constraints for binary outcomes.
result Allows for more precise estimation of monotonic functions in binary outcome models.
Estimates proper calibration errors and refinement terms in probabilistic predictions.
problem Lack of a general estimator for proper calibration errors and refinement terms with known statistical properties.
method Proposes a method for consistent, asymptotically unbiased estimation of proper calibration errors and refinement terms.
result Proves the relation between refinement and f-divergences, implying information monotonicity in neural networks.
Five simple soft sensor methodologies with two update conditions were compared on two experimentally-obtained datasets and one simulated dataset. The soft sensors investigated were moving window partial least squares regression (and a recursive variant), moving window random forest regression, the mean moving window of…
Improves k-NN for monotonic data with robustness against noise.
problem Class noise in real-life data violates monotonic constraints in k-NN.
method Monotonic Fuzzy k-NN (MonFkNN) with new fuzzy membership calculation.
result Significant accuracy improvements and robustness against monotonic noise.
We propose a data aggregation-based algorithm with monotonic convergence to a global optimum for a generalized version of the L1-norm error fitting model with an assumption of the fitting function. The proposed algorithm generalizes the recent algorithm in the literature, aggregate and iterative disaggregate (AID), whi…
We consider the problem of nonparametric regression when the covariate is d-dimensional, where d≥1. In this paper we introduce and study two nonparametric least squares estimators (LSEs) in this setting---the entirely monotonic LSE and the constrained Hardy-Krause variation LSE. We show that these two LSEs are…
Study examines explainable machine learning for monotonic models, finding Integrated gradients better for strong monotonicity.
problem Applying explainable machine learning to science-informed models.
method Proposed axioms for monotonicity, tested Shapley value and Integrated gradients methods.
result Integrated gradients provides better explanations for strong monotonicity.
This article studies the achievable guarantees on the error rates of certain learning algorithms, with particular focus on refining logarithmic factors. Many of the results are based on a general technique for obtaining bounds on the error rates of sample-consistent classifiers with monotonic error regions, in the real…
Near-optimal private tests for simple and MLR hypotheses developed under Gaussian differential privacy.
problem Developing private tests for simple and MLR hypotheses under Gaussian differential privacy.
method A private mean estimator with data-driven clamping bounds, constructing private test statistics.
result Private tests achieve the same asymptotic relative efficiency as non-private most powerful tests.
Formula proves monotonicity for anisotropic minimal hypersurfaces.
problem Understanding anisotropic minimal hypersurfaces.
method Proved a monotonicity formula under a sign assumption on the Minkowski norm.
result Monotonicity formula for anisotropic minimal hypersurfaces.
Scaling laws in linear regression explain model performance improvements with size and data.
problem Disagreement between empirical neural scaling laws and conventional wisdom on variance error.
method Infinite dimensional linear regression setup, one-pass SGD, Gaussian prior, power-law spectrum.
result Variance error is dominated by other errors, disappearing from the bound due to SGD's implicit regularization.
New example of manifolds with monotonic heat kernels found.
problem Understanding monotonicity of heat kernels on manifolds.
method Analyzing new examples and classifying flat tori.
result Generic metrics fail monotonicity at large times.
The paper develops algorithms to restore monotonicity in non-monotone functions.
problem Non-monotone solutions from heuristic algorithms need to be corrected.
method Develops algorithms to restore monotonicity with limited queries.
result Restores monotonicity while degrading the function value by at most ε.
Nonnegative matrix factorization (NMF) factorizes a non-negative matrix into product of two non-negative matrices, namely a signal matrix and a mixing matrix. NMF suffers from the scale and ordering ambiguities. Often, the source signals can be monotonous in nature. For example, in source separation problem, the source…
Unified model improves multi-task learning by accounting for temporal misalignment.
problem Poor predictive performance and uncertainty quantification due to temporal misalignment in multi-task learning.
method Uses Gaussian processes to model correlations and includes a monotonic warp of the input data to account for temporal misalignment.
result Improves predictive performance and uncertainty quantification in multi-task learning.