arXiv research

Paper compares hard and soft EM for BN learning from incomplete data.

Assignment methods are at the heart of many algorithms for unsupervised learning and clustering - in particular, the well-known K-means and Expectation-Maximization (EM) algorithms. In this work, we study several different methods of assignment, including the "hard" assignments used by K-means and the ?soft' assignment…

An evolutionary algorithm (EA) is developed as an alternative to the EM algorithm for parameter estimation in model-based clustering. This EA facilitates a different search of the fitness landscape, i.e., the likelihood surface, utilizing both crossover and mutation. Furthermore, this EA represents an efficient approac…

Optimizes clustering in Gaussian mixtures with varying covariance matrices.

In this paper, we formulate the problem of inferring a Finite Mixture Model from discrete data as an optimal transport problem with entropic regularization of parameter $λ\geq 0$. Our method unifies hard and soft clustering, the Expectation-Maximization (EM) algorithm being exactly recovered for $λ=1$. The family of cl…

We consider the problem of inference in a linear regression model in which the relative ordering of the input features and output labels is not known. Such datasets naturally arise from experiments in which the samples are shuffled or permuted during the protocol. In this work, we propose a framework that treats the un…

Nonlinearity is crucial to the performance of a deep (neural) network (DN). To date there has been little progress understanding the menagerie of available nonlinearities, but recently progress has been made on understanding the rôle played by piecewise affine and convex nonlinearities like the ReLU and absolute value …

Develops a high-dimensional differentially-private EM algorithm with near-optimal statistical guarantees.

In recent years there has been a flurry of works on learning Bayesian networks from data. One of the hard problems in this area is how to effectively learn the structure of a belief network from incomplete data- that is, in the presence of missing values or hidden variables. In a recent paper, I introduced an algorithm…

This paper is concerned with the hard thresholding operator which sets all but the $k$ largest absolute elements of a vector to zero. We establish a {\em tight} bound to quantitatively characterize the deviation of the thresholded solution from a given signal. Our theoretical result is universal in the sense that it ho…

Emerging market hard-currency bonds are an asset class of growing importance, and contain exposure to an EM sovereign and the underlying industry. The authors investigate how to model this as a modification of the well-known first-to-default (FtD) basket, using the structural model, and find the approach feasible.

OTSS learns personalized decision weights from logged decisions and outputs.

The study of higher-order homology embeddings for manifold topology.

This paper establishes a statistical versus computational trade-off for solving a basic high-dimensional machine learning problem via a basic convex relaxation method. Specifically, we consider the {\em Sparse Principal Component Analysis} (Sparse PCA) problem, and the family of {\em Sum-of-Squares} (SoS, aka Lasserre/…

We describe $k$-MLE, a fast and efficient local search algorithm for learning finite statistical mixtures of exponential families such as Gaussian mixture models. Mixture models are traditionally learned using the expectation-maximization (EM) soft clustering technique that monotonically increases the incomplete (expec…

Enhances large language models' reasoning through simpler off-policy reinforcement learning.

We give a polynomial-time algorithm for learning neural networks with one layer of sigmoids feeding into any Lipschitz, monotone activation function (e.g., sigmoid or ReLU). We make no assumptions on the structure of the network, and the algorithm succeeds with respect to {\em any} distribution on the unit ball in $n$ …

We derive a novel variational expectation maximization approach based on truncated posterior distributions. Truncated distributions are proportional to exact posteriors within subsets of a discrete state space and equal zero otherwise. The treatment of the distributions' subsets as variational parameters distinguishes …

Proposes new methods for Markov chain choice models with panel data.

Federated learning is viewed as a hierarchical latent variable model for new algorithm development.

New algorithm robustly estimates sparse models in high dimensions with corrupted data.

Iterative thresholding algorithms seek to optimize a differentiable objective function over a sparsity or rank constraint by alternating between gradient steps that reduce the objective, and thresholding steps that enforce the constraint. This work examines the choice of the thresholding operator, and asks whether it i…

Many real world problems can now be effectively solved using supervised machine learning. A major roadblock is often the lack of an adequate quantity of labeled data for training. A possible solution is to assign the task of labeling data to a crowd, and then infer the true label using aggregation methods. A well-known…

Crowdsourcing is a popular paradigm for effectively collecting labels at low cost. The Dawid-Skene estimator has been widely used for inferring the true labels from the noisy labels provided by non-expert crowdsourcing workers. However, since the estimator maximizes a non-convex log-likelihood function, it is hard to t…

We introduce 'mixed LICORS', an algorithm for learning nonlinear, high-dimensional dynamics from spatio-temporal data, suitable for both prediction and simulation. Mixed LICORS extends the recent LICORS algorithm (Goerg and Shalizi, 2012) from hard clustering of predictive distributions to a non-parametric, EM-like sof…

Learning general latent-variable probabilistic graphical models is a key theoretical challenge in machine learning and artificial intelligence. All previous methods, including the EM algorithm and the spectral algorithms, face severe limitations that largely restrict their applicability and affect their performance. In…

Paper tackles MLR prediction error without assuming realizable models.

Sparse activations in neural networks are hard to exploit but lead to advantages in learning.

Recently, supervised hashing methods have attracted much attention since they can optimize retrieval speed and storage cost while preserving semantic information. Because hashing codes learning is NP-hard, many methods resort to some form of relaxation technique. But the performance of these methods can easily deterior…

We study the problem of {\em properly} learning large margin halfspaces in the agnostic PAC model. In more detail, we study the complexity of properly learning $d$-dimensional halfspaces on the unit ball within misclassification error $α\cdot \mathrm{OPT}_γ + ε$, where $\mathrm{OPT}_γ$ is the optimal $γ$-margin error r…

Data-driven fault diagnostics and prognostics suffers from class-imbalance problem in industrial systems and it raises challenges to common machine learning algorithms as it becomes difficult to learn the features of the minority class samples. Synthetic oversampling methods are commonly used to tackle these problems b…

RL-LOW algorithm achieves exponential simple regret in offline RLHF with pairwise comparisons.

Physics-constrained neural nets solve EM fields of charged particle beams.

This paper poses some basic questions about instances (hard to find) of a special problem in 3-manifold topology. "Important though the general concepts and propositions may be with the modern industrious passion for axiomatizing and generalizing has presented us ... nevertheless I am convinced that the special problem…

We derive both {\em local} and {\em global} generalized {\em Bianchi identities} for classical Lagrangian field theories on gauge-natural bundles. We show that globally defined generalized Bianchi identities can be found without the {\em a priori} introduction of a connection. The proof is based on a {\em global} decom…

Paper proposes a faster SPIDER-EM variant for large-scale nonconvex optimization.

We consider the geometric formulation of the Hamiltonian formalism for field theory in terms of {\em Hamiltonian connections} and {\em multisymplectic forms}. In this framework the covariant Hamilton equations for Mechanics and field theory are defined in terms of multisymplectic $(n+2)$--forms, where $n$ is the dimens…

Optimizing over the set of orthogonal matrices is a central component in problems like sparse-PCA or tensor decomposition. Unfortunately, such optimization is hard since simple operations on orthogonal matrices easily break orthogonality, and correcting orthogonality usually costs a large amount of computation. Here we…

The EM algorithm is one of many important tools in the field of statistics. While often used for imputing missing data, its widespread applications include other common statistical tasks, such as clustering. In clustering, the EM algorithm assumes a parametric distribution for the clusters, whose parameters are estimat…

Bayesian networks (BN) are used in a big range of applications but they have one issue concerning parameter learning. In real application, training data are always incomplete or some nodes are hidden. To deal with this problem many learning parameter algorithms are suggested foreground EM, Gibbs sampling and RBE algori…

We consider the Nordic electricity spot market from mid 1992 to the end of year 2000. This market is found to be well approximated by an anti-persistent self-affine (mean-reverting) walk. It is characterized by a Hurst exponent of $H\simeq 0.41$ over three orders of magnitude in time ranging from days to years. We argu…

Gradient EM converges globally for over-parameterized Gaussian mixtures.

EM algorithm converges in KL divergence for exponential families via mirror descent.

Improves EM algorithm for better local optima in mixture models.

The study characterizes Hermitian manifolds with parallel Bismut-Strominger torsion.

sEM uses optimal transport to improve EM algorithm for better convergence and avoiding local optima.

Paper analyzes EM algorithm's trajectory in 2MLR, revealing cycloid behavior.

A new EM gradient algorithm for mixture models with skewed components.