arXiv research

We study losses for binary classification and class probability estimation and extend the understanding of them from margin losses to general composite losses which are the composition of a proper loss with a link function. We characterise when margin losses can be proper composite losses, explicitly show how to determ…

We consider composite loss functions for multiclass prediction comprising a proper (i.e., Fisher-consistent) loss over probability distributions and an inverse link function. We establish conditions for their (strong) convexity and explore the implications. We also show how the separation of concerns afforded by using …

Edgeworth Accountant calculates privacy loss under differential privacy compositions efficiently.

Paper simplifies DP composition for adaptive privacy budgets, enabling better privacy and accuracy in deep learning.

New approach for distributed online optimization of non-convex losses with sublinear regret.

Paper introduces arctan pinball loss for XGBoost quantile regression.

This paper advances FL algorithms for composite optimization and statistical recovery.

New filters match advanced composition for adaptive privacy, with practical constants.

Paper extends FFT-based differential privacy method to heterogeneous compositions.

The goal of online prediction with expert advice is to find a decision strategy which will perform almost as well as the best expert in a given pool of experts, on any sequence of outcomes. This problem has been widely studied and $O(\sqrt{T})$ and $O(\log{T})$ regret bounds can be achieved for convex losses (\cite{zin…

AutoBayes simplifies variational inference by composing models and optimizing them.

A new privacy accountant for Gaussian differential privacy measures individual privacy losses.

Since its inception, the modus operandi of multi-task learning (MTL) has been to minimize the task-wise mean of the empirical risks. We introduce a generalized loss-compositional paradigm for MTL that includes a spectrum of formulations as a subfamily. One endpoint of this spectrum is minimax MTL: a new MTL formulation…

Study non-oblivious adversarial bandits with delayed feedback and propose algorithms with improved regret bounds.

Extends knockoff filter for composite null hypotheses in variable selection.

Generalization error defines the discriminability and the representation power of a deep model. In this work, we claim that feature space design using deep compositional function plays a significant role in generalization along with explicit and implicit regularizations. Our claims are being established with several im…

The study of a machine learning problem is in many ways is difficult to separate from the study of the loss function being used. One avenue of inquiry has been to look at these loss functions in terms of their properties as scoring rules via the proper-composite representation, in which predictions are mapped to probab…

Improved subgradient method tackles ill-conditioned composite optimization problems.

The intuition of risk is based on two main concepts: loss and variability. In this paper, we present a composition of risk and deviation measures, which contemplate these two concepts. Based on the proposed Limitedness axiom, we prove that this resulting composition, based on properties of the two components, is a cohe…

Inverse depth scaling found in LLMs due to similar layers averaging error.

HydaLearn dynamically adjusts task weights for better MTL performance.

Proposes a new loss function for robust learning.

New kernels from neural networks show better performance than traditional methods.

New method tackles composite optimization with error feedback.

The paper explores using historical data to improve clinical trial analysis by optimizing covariate weights.

Simple model outperforms neural networks on language understanding tasks.

Improved online learning with time-varying constraints for complex domains.

We propose a robust inferential procedure for assessing uncertainties of parameter estimation in high-dimensional linear models, where the dimension $p$ can grow exponentially fast with the sample size $n$. Our method combines the de-biasing technique with the composite quantile function to construct an estimator that …

A graph neural network improves multivariate post-processing of ensemble forecasts.

New method controls posterior collapse in VAEs without network architecture constraints.

The problem of bipartite ranking, where instances are labeled positive or negative and the goal is to learn a scoring function that minimizes the probability of mis-ranking a pair of positive and negative instances (or equivalently, that maximizes the area under the ROC curve), has been widely studied in recent years. …

This paper analyzes how diffusion models learn and generalize concepts.

Traditional dictionary learning methods are based on quadratic convex loss function and thus are sensitive to outliers. In this paper, we propose a generic framework for robust dictionary learning based on concave losses. We provide results on composition of concave functions, notably regarding super-gradient computati…

Paper improves privacy bounds for shuffle model using novel numerical techniques.

Novel oracle-type inequality for logistic loss in DNNs achieves sharp convergence rates.

This research tackles group fairness in predictive process monitoring by ensuring predictions are independent of sensitive group membership.

New algorithm for federated learning with non-smooth regularizers.

Network embedding has proved extremely useful in a variety of network analysis tasks such as node classification, link prediction, and network visualization. Almost all the existing network embedding methods learn to map the node IDs to their corresponding node embeddings. This design principle, however, hinders the ex…

We provide novel theoretical results regarding local optima of regularized $M$-estimators, allowing for nonconvexity in both loss and penalty functions. Under restricted strong convexity on the loss and suitable regularity conditions on the penalty, we prove that \emph{any stationary point} of the composite objective f…

We introduce a scalable measure of curvature for analyzing training dynamics of large language models.

Unified analysis of stochastic gradient methods for convex and smooth optimization.

Generative processes in biology and other fields often produce data that can be regarded as resulting from a composition of basic features. Here we present an unsupervised method based on autoencoders for inferring these basic features of data. The main novelty in our approach is that the training is based on the optim…

A composite loss framework is proposed for low-rank modeling of data consisting of interesting and common values, such as excess zeros or missing values. The methodology is motivated by the generalized low-rank framework and the hurdle method which is commonly used to analyze zero-inflated counts. The model is demonstr…

Study feature representations induced by dependence between variables.

AutoScale improves LLM pre-training by adjusting data mixtures at different scales.

Paper proposes fitting loss functions to data using source functions from information geometry.

Deep-HGP uses Bayesian nonparametric approach for complex data regression.

New method shows hidden state can significantly improve differential privacy in SGD.