arXiv research

Sampling is a fundamental technique, and sampling without replacement is often desirable when duplicate samples are not beneficial. Within machine learning, sampling is useful for generating diverse outputs from a trained model. We present an elegant procedure for sampling without replacement from a broad class of rand…

SGD without replacement decouples into curvature-following and flatness-regularizing steps.

We propose a generative model for adversarial attack. The model generates subtle but predictive patterns from the input. To perform an attack, it replaces the patterns of the input with those generated based on examples from some other class. We demonstrate our model by attacking CNN on MNIST.

Spatial graphs study tangle replacement with equivalence classes.

We introduce new symplectic cut-and-paste operations that generalize the rational blowdown. In particular, we will define $k$-replaceable plumbings to be those that, heuristically, can be symplectically replaced by Euler characteristic $k$ 4-manifolds. We will then classify 2-replaceable linear plumbings, construct 2-r…

The well-known Gumbel-Max trick for sampling from a categorical distribution can be extended to sample $k$ elements without replacement. We show how to implicitly apply this 'Gumbel-Top-$k$' trick on a factorized distribution over sequences, allowing to draw exact samples without replacement using a Stochastic Beam Sea…

This paper addresses credit valuation adjustment with a new closeout convention.

Develops algorithms to optimize machine replacement schedules using operational data.

Tangle replacements help in understanding knot properties.

Stochastic gradient descent without replacement sampling is widely used in practice for model training. However, the vast majority of SGD analyses assumes data is sampled with replacement, and when the function minimized is strongly convex, an $\mathcal{O}\left(\frac{1}{T}\right)$ rate can be established when SGD is ru…

New method for simplifying knots with specific properties.

Any autoencoder network can be turned into a generative model by imposing an arbitrary prior distribution on its hidden code vector. Variational Autoencoder (VAE) [2] uses a KL divergence penalty to impose the prior, whereas Adversarial Autoencoder (AAE) [1] uses {\it generative adversarial networks} GAN [3]. GAN trade…

Regularization is an important component of predictive model building. The hybrid bootstrap is a regularization technique that functions similarly to dropout except that features are resampled from other training points rather than replaced with zeros. We show that the hybrid bootstrap offers superior performance to dr…

If M and N are equivariantly homotopy equivalent G-manifolds, then the fixed sets M^G and N^G are also homotopy equivalent. The replacement problem asks the converse question: If F is homotopy equivalent to the fixed set M^G, is F = N^G for a G-manifold equivariantly homotopy equivalent to M? We prove that for locally …

Sampling without replacement speeds up optimization in minimax problems.

Differential privacy is a useful tool to build machine learning models which do not release too much information about the training data. We study the Rényi differential privacy of stochastic gradient descent when each training example is sampled without replacement (also known as cyclic SGD). Cyclic SGD is typically f…

TROLL improves RL for LLMs by replacing clipping with a trust region projection.

Dedicated to Professor Gromoll: The aim of our article is to generalize the Toponogov comparison theorem to a complete Riemannian manifold with smooth convex boundary. A geodesic triangle will be replaced by an open (geodesic) triangle standing on the boundary of the manifold, and a model surface will be replaced by th…

New research disproves a key conjecture in optimization.

New sketches for weighted $\ell_p$ sampling without replacement improve accuracy and efficiency.

The paper develops methods to estimate POMDPs from partial information.

The multimodal web elements such as text and images are associated with inherent memory costs to store and transfer over the Internet. With the limited network connectivity in developing countries, webpage rendering gets delayed in the presence of high-memory demanding elements such as images (relative to text). To ove…

We study stochastic gradient descent {\em without replacement} (\sgdwor) for smooth convex functions. \sgdwor is widely observed to converge faster than true \sgd where each sample is drawn independently {\em with replacement} \cite{bottou2009curiously} and hence, is more popular in practice. But it's convergence prope…

We develop an analog of harmonic replacement in the gauge theory context. The idea behind harmonic replacement dates back to Schwarz and Perron. The technique, as introduced by Jost and further developed by Colding and Minicozzi, involves taking a map $v\colonΣ\to M$ defined on a surface $Σ$ and replacing its values on…

Optimal algorithms for online convex optimization with random order.

Differentiable pipeline replaces non-differentiable CAE components for shape optimization.

Rank minimization has attracted a lot of attention due to its robustness in data recovery. To overcome the computational difficulty, rank is often replaced with nuclear norm. For several rank minimization problems, such a replacement has been theoretically proven to be valid, i.e., the solution to nuclear norm minimiza…

Randomized algorithms that base iteration-level decisions on samples from some pool are ubiquitous in machine learning and optimization. Examples include stochastic gradient descent and randomized coordinate descent. This paper makes progress at theoretically evaluating the difference in performance between sampling wi…

Stochastic gradient methods for machine learning and optimization problems are usually analyzed assuming data points are sampled \emph{with} replacement. In practice, however, sampling \emph{without} replacement is very common, easier to implement in many cases, and often performs better. In this paper, we provide comp…

Boring is an operation which converts a knot or two-component link in a 3--manifold into another knot or two-component link. It generalizes rational tangle replacement and can be described as a type of 2--handle attachment. Sutured manifold theory is used to study the existence of essential spheres and planar surfaces …

Recently we generalized Toponogov's comparison theorem to a complete Riemannian manifold with smooth convex boundary, where a geodesic triangle was replaced by an open (geodesic) triangle standing on the boundary of the manifold, and a model surface was replaced by the universal covering surface of a cylinder of revolu…

Improved convergence for VIPs with SEG-RR, a variant of SEG with random reshuffling.

The key to a Transformer model is the self-attention mechanism, which allows the model to analyze an entire sequence in a computationally efficient manner. Recent work has suggested the possibility that general attention mechanisms used by RNNs could be replaced by active-memory mechanisms. In this work, we evaluate wh…

Study bounds the Morse index of a special torus to 1.

Skeleta and other pure subsets of manifold stratified spaces are shown to have neighborhoods which are teardrops of stratified approximate fibrations (under dimension and compactness assumptions). In general, the stratified approximate fibrations cannot be replaced by bundles, and the teardrops cannot be replaced by ma…

Crowdsourcing platforms are now extensively used for conducting subjective pairwise comparison studies. In this setting, a pairwise comparison dataset is typically gathered via random sampling, either \emph{with} or \emph{without} replacement. In this paper, we use tools from random graph theory to analyze these two ra…

This research proposes the econophysics kinetic market model as an evolutionary algorithm's instance. The immediate results from this proposal is a new replacement rule for family competition genetic algorithms. It also represents a starting point to adding evolvable entities to kinetic market models.

A frame independent formulation of analytical mechanics in the Newtonian space-time is presented. The differential geometry of affine values i.e., the differential geometry in which affine bundles replace vector bundles and sections of one dimensional affine bundles replace functions on manifolds, is used. Lagrangian a…

The paper introduces methods to quantify uncertainty in sampling without replacement.

The goal of few-shot learning is to learn a model that can recognize novel classes based on one or few training data. It is challenging mainly due to two aspects: (1) it lacks good feature representation of novel classes; (2) a few of labeled data could not accurately represent the true data distribution and thus it's …

Parrot learns optimal cache replacement policies using imitation learning.

Replacing Black-Scholes' driving process, Brownian motion, with fractional Brownian motion allows for incorporation of a past dependency of stock prices but faces a few major downfalls, including the occurrence of arbitrage when implemented in the financial market. We present the development, testing, and implementatio…

This paper studies the convergence behaviour of dictionary learning via the Iterative Thresholding and K-residual Means (ITKrM) algorithm. On one hand it is proved that ITKrM is a contraction under much more relaxed conditions than previously necessary. On the other hand it is shown that there seem to exist stable fixe…

Let (M,g) be a smooth compact Riemannian manifold without boundary of dimension n>=6. We prove that {align*} \|u\|_{L^{2^*}(M,g)}^2 \le K^2\int_M\{|\nabla_g u|^2+c(n)R_gu^2\}dv_g +A\|u\|_{L^{2n/(n+2)}(M,g)}^2, {align*} for all u\in H^1(M), where 2^*=2n/(n-2), c(n)=(n-2)/[4(n-1)], R_g is the scalar curvature, $K^{-1}=\i…

We present an alternative layer to convolution layers in convolutional neural networks (CNNs). Our approach reduces the complexity of convolutions by replacing it with binary decisions. Those binary decisions are used as indexes to conditional distributions where each weight represents a leaf in a decision tree. This m…

We perform a replacement procedure in order to produce a free boundary minimal surface whose area achieves the min-max value over all disk sweepouts of a manifold whose boundary lie in a submanifold. Our result is based on a proof of the convexity of the energy for free boundary harmonic maps and a generalization of Co…

Normalizing flows attempt to model an arbitrary probability distribution through a set of invertible mappings. These transformations are required to achieve a tractable Jacobian determinant that can be used in high-dimensional scenarios. The first normalizing flow designs used coupling layer mappings built upon affine …

We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization. Similarly, the logarithm in the log loss we use for training is replaced by a low temperature logarithm. By tuning the two temperatures we create loss functions that are non…