arXiv research

This work is an analytical and numerical study of the composition of several fractals into one and of the relation between the composite dimension and the dimensions of the component fractals. In the case of composition of standard IFS with segments of equal size, the composite dimension can be expressed as a function …

Study on deep neural networks using branching processes and Mehler's formula.

The paper defines functions that induce bounded composition operators on RKHSs with analytic positive definite functions.

The stochastic gradient descent has been widely used for solving composite optimization problems in big data analyses. Many algorithms and convergence properties have been developed. The composite functions were convex primarily and gradually nonconvex composite functions have been adopted to obtain more desirable prop…

New algorithm reduces complexity for optimizing complex machine learning tasks.

This paper explores optimising acquisition functions in Bayesian optimisation.

Deep networks learn sparse hierarchical features without CoD.

Improved subgradient method tackles ill-conditioned composite optimization problems.

Gaussian processes struggle with compositional functions, but deep Gaussian processes can outperform.

Many machine learning, statistical inference, and portfolio optimization problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG)…

C-ADAM is a new adaptive solver for complex nested problems.

In this paper, we consider the convex and non-convex composition problem with the structure $\frac{1}{n}\sum\nolimits_{i = 1}^n {{F_i}( {G( x )} )}$, where $G( x )=\frac{1}{n}\sum\nolimits_{j = 1}^n {{G_j}( x )}$ is the inner function, and $F_i(\cdot)$ is the outer function. We explore the variance reduction based met…

Deep neural networks can approximate complex functions through repeated compositions of a fixed-size ReLU network.

We consider composite loss functions for multiclass prediction comprising a proper (i.e., Fisher-consistent) loss over probability distributions and an inverse link function. We establish conditions for their (strong) convexity and explore the implications. We also show how the separation of concerns afforded by using …

We study losses for binary classification and class probability estimation and extend the understanding of them from margin losses to general composite losses which are the composition of a proper loss with a link function. We characterise when margin losses can be proper composite losses, explicitly show how to determ…

Paper proposes iLPA for solving DC composite optimization problems, with applications to matrix completion with outliers.

New framework infers multiple classes per image for one-shot learning.

We consider the problem of minimizing the composition of a smooth (nonconvex) function and a smooth vector mapping, where the inner mapping is in the form of an expectation over some random variable or a finite sum. We propose a stochastic composite gradient method that employs an incremental variance-reduced estimator…

Improved Q-learning for robot control with faster data-efficiency.

DGPs collapse to near-deterministic transformations, limiting their compositional structure discovery.

We consider the composition optimization with two expected-value functions in the form of $\frac{1}{n}\sum\nolimits_{i = 1}^n F_i(\frac{1}{m}\sum\nolimits_{j = 1}^m G_j(x))+R(x)$, { which formulates many important problems in statistical learning and machine learning such as solving Bellman equations in reinforcement l…

Deep learning can learn compositional functions more efficiently by breaking them into stages.

Sublinearly structured DNNs achieve feature learning consistency for compositional functions.

Defines 'nowhere coexpanding functions' and studies their fixed points.

Here we study non-convex composite optimization: first, a finite-sum of smooth but non-convex functions, and second, a general function that admits a simple proximal mapping. Most research on stochastic methods for composite optimization assumes convexity or strong convexity of each function. In this paper, we extend t…

Unified algorithm for minimizing composite functions with flexible design.

New geometric approach for analyzing compositional data like gut microbiomes.

Unified analysis of multi-task functional linear regression with manifold and composite penalties.

Model predicts composite structures assembly quality with input uncertainty.

We consider optimization of composite objective functions, i.e., of the form $f(x)=g(h(x))$, where $h$ is a black-box derivative-free expensive-to-evaluate function with vector-valued outputs, and $g$ is a cheap-to-evaluate real-valued function. While these problems can be solved with standard Bayesian optimization, we…

Consider the stochastic composition optimization problem where the objective is a composition of two expected-value functions. We propose a new stochastic first-order method, namely the accelerated stochastic compositional proximal gradient (ASC-PG) method, which updates based on queries to the sampling oracle using tw…

BOIS optimizes complex systems by leveraging structural knowledge.

This work proves composite neural networks outperform their components under certain conditions.

New method for causal inference with complex treatment compositions.

In this paper we determine a number of meaningful compositions of higher order of a set of functions, which is considered in Malesevic (1998), in implicit and explicit form. Results which are obtained are applied to the vector analysis in order to determine the number of meaningful differential operations of higher ord…

Composite neural network improves PM2.5 prediction.

Curvature penalties improve interpretability of KANs without sacrificing accuracy.

We consider in this work a system of two stochastic differential equations named the perturbed compositional gradient flow. By introducing a separation of fast and slow scales of the two equations, we show that the limit of the slow motion is given by an averaged ordinary differential equation. We then demonstrate that…

Classical stochastic gradient methods are well suited for minimizing expected-value objective functions. However, they do not apply to the minimization of a nonlinear function involving expected values or a composition of two expected-value functions, i.e., problems of the form $\min_x \mathbf{E}_v [f_v\big(\mathbf{E}_…

New samplers improve compositional generation with diffusion models.

New sparse GP model learns compositional kernels efficiently.

New algorithm samples efficiently from complex composite potentials.

In this contribution we describe an approach to evolve composite covariance functions for Gaussian processes using genetic programming. A critical aspect of Gaussian processes and similar kernel-based models such as SVM is, that the covariance function should be adapted to the modeled data. Frequently, the squared expo…

Paper analyzes stability and generalization of SCO algorithms.

Study dual representations for quasiconvex systemic risk measures.

Transformer architecture struggles with complex tasks due to limitations in function composition.

This paper explores the non-convex composition optimization in the form including inner and outer finite-sum functions with a large number of component functions. This problem arises in some important applications such as nonlinear embedding and reinforcement learning. Although existing approaches such as stochastic gr…

Lower bounds set for infinite-precision transformers.