arXiv research

Paper proves method for calculating NML code length works for continuous models.

New coding theorem shows achievable rate matches theoretical limit.

Likelihood-based generative models are the backbones of lossless compression due to the guaranteed existence of codes with lengths close to negative log likelihood. However, there is no guaranteed existence of computationally efficient codes that achieve these lengths, and coding algorithms must be hand-tailored to spe…

Study of manifolds with prime cyclic group actions and curvature properties.

We introduce new definitions of universal and superuniversal computable codes, which are based on a code's ability to approximate Kolmogorov complexity within the prescribed margin for all individual sequences from a given set. Such sets of sequences may be singled out almost surely with respect to certain probability …

New quantum codes improve error correction with local tests.

For reliable transmission across a noisy communication channel, classical results from information theory show that it is asymptotically optimal to separate out the source and channel coding processes. However, this decomposition can fall short in the finite bit-length regime, as it requires non-trivial tuning of hand-…

ICQ improves high-dimensional similarity search without sacrificing precision.

Paper trains a Transformer to add numbers of any length.

We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our interest is in describing their convergence with sequence length, assuming no limit…

The Minimum Description Length (MDL) principle states that the optimal model for a given data set is that which compresses it best. Due to practial limitations the model can be restricted to a class such as linear regression models, which we address in this study. As in other formulations such as the LASSO and forward …

Constrained sequence codes have been widely used in modern communication and data storage systems. Sequences encoded with constrained sequence codes satisfy constraints imposed by the physical channel, hence enabling efficient and reliable transmission of coded symbols. Traditional encoding and decoding of constrained …

DeepJSCC-f uses feedback to improve image transmission quality.

Paper reinterprets majorizing measure theorem in terms of coding theory.

The design of codes for communicating reliably over a statistically well defined channel is an important endeavor involving deep mathematical research and wide-ranging practical applications. In this work, we present the first family of codes obtained via deep learning, which significantly beats state-of-the-art codes …

This paper addresses the nearest neighbor search problem under inner product similarity and introduces a compact code-based approach. The idea is to approximate a vector using the composition of several elements selected from a source dictionary and to represent this vector by a short code composed of the indices of th…

Non-parametric estimators improve quickest changepoint detection under irregular sequence lengths.

Suppose $C$ is a compact, $n$-edged two-cell of the centered dual decomposition of a locally finite set in the hyperbolic plane, a coarsening of the Delaunay tessellation which was introduced in the author's prior work. We describe an effectively computable lower bound on the area of $C$, given an $n$-tuple of positive…

This paper proposes a novel deep learning-based error correction coding scheme for AWGN channels under the constraint of one-bit quantization in the receivers. Specifically, it is first shown that the optimum error correction code that minimizes the probability of bit error can be obtained by perfectly training a speci…

Reasoning models generate differently based on problem difficulty, not just length.

Study on reducing forgetting in neural networks using compression theory.

End-to-end learning of codes for secure BPSK communication in Gaussian wiretap channel.

Coding theory is a central discipline underpinning wireline and wireless modems that are the workhorses of the information age. Progress in coding theory is largely driven by individual human ingenuity with sporadic breakthroughs over the past century. In this paper we study whether it is possible to automate the disco…

We introduce Compositional Imitation Learning and Execution (CompILE): a framework for learning reusable, variable-length segments of hierarchically-structured behavior from demonstration data. CompILE uses a novel unsupervised, fully-differentiable sequence segmentation module to learn latent encodings of sequential d…

Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimiz…

We present a neural model for representing snippets of code as continuous distributed vectors ("code embeddings"). The main idea is to represent a code snippet as a single fixed-length $\textit{code vector}$, which can be used to predict semantic properties of the snippet. This is performed by decomposing code to a col…

The article shows how to count small eigenvalues without assuming Morse functions.

A new algorithm reduces regret in multi-player bandits without collision info.

This work challenges the assumption that shorter conformal prediction intervals are always better.

Shorter adversarial prompts help protect LLMs from jailbreak attacks.

Model learns brevity by exposing to easy problems, improving efficiency without explicit length penalties.

Method generates adversarial examples to improve classifier robustness.

Paper proposes IABF to improve NECST robustness.

Paper uses RNNs to design LDPC codes for binary erasure channels.

Effective representation learning of electronic health records is a challenging task and is becoming more important as the availability of such data is becoming pervasive. The data contained in these records are irregular and contain multiple modalities such as notes, and medical codes. They are preempted by medical co…

We develop a Bayesian approach to learning from sequential data by using Gaussian processes (GPs) with so-called signature kernels as covariance functions. This allows to make sequences of different length comparable and to rely on strong theoretical results from stochastic analysis. Signatures capture sequential struc…

We present LumièreNet, a simple, modular, and completely deep-learning based architecture that synthesizes, high quality, full-pose headshot lecture videos from instructor's new audio narration of any length. Unlike prior works, LumièreNet is entirely composed of trainable neural network modules to learn mapping functi…

New algorithm for multi-player bandits with collision-dependent rewards.

Recent developments in the field of deep learning have motivated many researchers to apply these methods to problems in quantum information. Torlai and Melko first proposed a decoder for surface codes based on neural networks. Since then, many other researchers have applied neural networks to study a variety of problem…

Motion Code models time series dynamics with sparse approximations.

We consider the problem of generating automatic code given sample input-output pairs. We train a neural network to map from the current state and the outputs to the program's next statement. The neural network optimizes multiple tasks concurrently: the next operation out of a set of high level commands, the operands of…

The minimum message length principle is an information theoretic criterion that links data compression with statistical inference. This paper studies the strict minimum message length (SMML) estimator for $d$-dimensional exponential families with continuous sufficient statistics, for all $d \ge 1$. The partition of an …

Transformer struggles with arithmetic length but improves with explicit structure encoding.

We present new computations of approximately length-minimizing polygons with fixed thickness. These curves model the centerlines of "tight" knotted tubes with minimal length and fixed circular cross-section. Our curves approximately minimize the ropelength (or quotient of length and thickness) for polygons in their kno…

Meta-learning framework improves short utterance speaker recognition.

New method improves bivariate causal discovery by accurately estimating cause variable complexity.

Measures neural network information transfer for generalization.

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-le…