Machine learning algorithms are typically run on large scale, distributed compute infrastructure that routinely face a number of unavailabilities such as failures and temporary slowdowns. Adding redundant computations using coding-theoretic tools called "codes" is an emerging technique to alleviate the adverse effects …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Coded Computing”
We introduce new definitions of universal and superuniversal computable codes, which are based on a code's ability to approximate Kolmogorov complexity within the prescribed margin for all individual sequences from a given set. Such sets of sequences may be singled out almost surely with respect to certain probability …
Coded computation techniques provide robustness against straggling servers in distributed computing, with the following limitations: First, they increase decoding complexity. Second, they ignore computations carried out by straggling servers; and they are typically designed to recover the full gradient, and thus, canno…
Matrix multiplication is a fundamental building block for large scale computations arising in various applications, including machine learning. There has been significant recent interest in using coding to speed up distributed matrix multiplication, that are robust to stragglers (i.e., machines that may perform slower …
Gradient coding is a technique for straggler mitigation in distributed learning. In this paper we design novel gradient codes using tools from classical coding theory, namely, cyclic MDS codes, which compare favorably with existing solutions, both in the applicable range of parameters and in the complexity of the invol…
Gradient codes adapt to varying straggler counts in distributed learning.
problem Mitigating slow worker nodes (stragglers) in distributed machine learning.
method Proposes a flexible gradient coding scheme that concatenates codes for different straggler tolerances, adapting to actual straggler counts.
result Significantly lower latency compared to fixed-tolerance gradient codes.
This paper develops coding techniques to reduce the running time of distributed learning tasks. It characterizes the fundamental tradeoff to compute gradients (and more generally vector summations) in terms of three parameters: computation load, straggler tolerance and communication cost. It further gives an explicit c…
Fault-tolerant neural networks inspired by biological error correction codes.
problem Achieving reliable computation with unreliable neurons.
method Using biological error correction codes from grid cells in the mammalian cortex to develop a fault-tolerant neural network.
result Noisy biological neurons operate below a fault-tolerance threshold, suggesting a mechanism for reliable computation in the brain.
Distributed algorithms are often beset by the straggler effect, where the slowest compute nodes in the system dictate the overall running time. Coding-theoretic techniques have been recently proposed to mitigate stragglers via algorithmic redundancy. Prior work in coded computation and gradient coding has mainly focuse…
Coded Federated Learning speeds up training in edge computing networks.
problem Slow convergence in Federated Learning due to heterogeneity and stochastic fluctuations.
method Exploiting statistical properties of compute and communication delays, distributed kernel embedding, and random Fourier features.
result Significant performance gains for CodedFedL in distributed non-linear regression and classification problems.
Distributed implementations of gradient-based methods, wherein a server distributes gradient computations across worker machines, suffer from slow running machines, called 'stragglers'. Gradient coding is a coding-theoretic framework to mitigate stragglers by enabling the server to recover the gradient sum in the prese…
Generative AI decodes quantum codes without labeled data.
problem Efficient decoding of quantum error-correcting codes.
method Generative Transformers learn logical operators from unsupervised syndromes.
result Significantly better decoding accuracy than traditional methods.
CodedFedL speeds up federated learning in MEC networks by 15x.
problem Slow convergence in federated learning due to heterogeneity and stochastic fluctuations.
method Injects structured coding redundancy into federated learning to mitigate stragglers and speed up training.
result CodedFedL speeds up the training procedure by up to 15x compared to benchmark schemes.
Transformer models waste resources on long-context tasks.
problem Redundant attention computations in Transformer models for long-context tasks.
method Reformulate sequence modeling as supervised learning, analyze attention sparsity, formulate attention optimization as linear coding problem, propose Dynamic Group Attention.
result DGA reduces computational costs while maintaining performance.
Machine learning models that take computer program source code as input typically use Natural Language Processing (NLP) techniques. However, a major challenge is that code is written using an open, rapidly changing vocabulary due to, e.g., the coinage of new variable and method names. Reasoning over such a vocabulary i…
Khovanov homology helps create quantum error-correcting codes.
problem Creating robust quantum error-correcting codes.
method Using Khovanov homology and its extensions to define and analyze quantum codes.
result New families of quantum codes with desirable properties.
This paper considers the problem of implementing large-scale gradient descent algorithms in a distributed computing setting in the presence of {\em straggling} processors. To mitigate the effect of the stragglers, it has been previously proposed to encode the data with an erasure-correcting code and decode at the maste…
Gradient descent and its many variants, including mini-batch stochastic gradient descent, form the algorithmic foundation of modern large-scale machine learning. Due to the size and scale of modern data, gradient computations are often distributed across multiple compute nodes. Unfortunately, such distributed implement…
Sparse codes improve optimal control tasks with correlated inputs.
problem Optimal control tasks with correlated feature inputs.
method Used a sparse code to represent natural images in an optimal control task solved with neuro-dynamic programming.
result An over-complete sparse code increases memory capacity and learning speed beyond a complete code.
Towards Run Time Estimation of the Gaussian Chemistry Code for SEAGrid Science Gatewayphysics.comp-ph
Accurate estimation of the run time of computational codes has a number of significant advantages for scientific computing. It is required information for optimal resource allocation, improving turnaround times and utilization of science gateways. Furthermore, it allows users to better plan and schedule their research,…
Topological theory for qLDPC codes enables non-Clifford gates and magic state injection.
problem Fault-tolerant quantum computation in qLDPC codes with non-Clifford gates and magic state resources.
method Developed a topological theory using simplicial or CW complex structures and deformation retraction.
result Achieved non-Clifford gates and magic state injection in qLDPC codes with constant rate and polynomial distance.
CORAL: COde RepresentAtion Learning with Weakly-Supervised Transformers for Analyzing Data Analysiscs.LG
Model learns code representations from comments for data analysis tasks.
problem Lack of descriptive labels for analyzing large code corpora.
method Weakly supervised transformer architecture for joint code and comment representation.
result Model achieves 38% accuracy increase over expert-supplied heuristics.
Code-switching, the alternation of languages within a conversation or utterance, is a common communicative phenomenon that occurs in multilingual communities across the world. This survey reviews computational approaches for code-switched Speech and Natural Language Processing. We motivate why processing code-switched …
New fault-tolerant quantum gates for homological LDPC codes with constant or almost-constant rate.
problem Fault-tolerant quantum computing for homological LDPC codes with constant or almost-constant encoding rate.
method Derive generic formula for transversal and logical gates acting on 3-manifolds, using higher symmetries and cup product cohomology.
result Parallelizable logical gates for homological LDPC codes with constant or almost-constant rate.
New method calculates knot and link properties using state codes.
problem Determining the unoriented genus and crosscap number of prime alternating knots and links.
method Encoding states as tuples and using them to compute genus and crosscap number.
result Computed values for all such links through 14 crossings and knots through 19 crossings, identifying patterns.
For reliable transmission across a noisy communication channel, classical results from information theory show that it is asymptotically optimal to separate out the source and channel coding processes. However, this decomposition can fall short in the finite bit-length regime, as it requires non-trivial tuning of hand-…
We study the problem of multivariate regression where the data are naturally grouped, and a regression matrix is to be estimated for each group. We propose an approach in which a dictionary of low rank parameter matrices is estimated across groups, and a sparse linear combination of the dictionary elements is estimated…
New metric space for ReLU codes connects to network safety and robustness.
problem Lack of metrics capturing network safety and robustness beyond accuracy.
method Introduces a metric space of ReLU activation codes with a truncated Hamming distance.
result Establishes an isometry between ReLU codes and polyhedral bodies related to safety and robustness.
GKP codes connect quantum gates to algebraic curves, enabling fault-tolerant quantum computation.
problem Implementing fault-tolerant quantum computation in quantum harmonic oscillator systems.
method Exploring the topological and algebraic structure of GKP codes, showing how gates correspond to symplectic automorphisms and mapping class groups of surfaces.
result GKP Clifford gates are identified with symplectic automorphisms of GKP lattices and mapping class groups of surfaces, providing a topological interpretation of fault tolerance.
Similarity search retrieves the nearest neighbors of a query vector from a dataset of high-dimensional vectors. As the size of the dataset grows, the cost of performing the distance computations needed to implement a query can become prohibitive. A method often used to reduce this computational cost is quantization of …
The paper relaxes constraints on predictive coding models, making them more biologically plausible.
problem Neurophysiological models of predictive coding are not fully biologically plausible.
method The paper relaxes constraints on standard predictive coding algorithms by removing neurally implausible features.
result The removal of neurally implausible features does not significantly affect learning performance.
We address the problem of any-code completion - generating a missing piece of source code in a given program without any restriction on the vocabulary or structure. We introduce a new approach to any-code completion that leverages the strict syntax of programming languages to model a code snippet as a tree - structural…
Likelihood-based generative models are the backbones of lossless compression due to the guaranteed existence of codes with lengths close to negative log likelihood. However, there is no guaranteed existence of computationally efficient codes that achieve these lengths, and coding algorithms must be hand-tailored to spe…
STRATA generates code adversarial examples efficiently without gradients.
problem Generating adversarial examples for code that retains functional meaning.
method Uses token frequency statistics to construct gradient-free adversarial examples.
result Empirically outperforms gradient-based methods with less information and effort.
Sparse coding has been popularly used as an effective data representation method in various applications, such as computer vision, medical imaging and bioinformatics, etc. However, the conventional sparse coding algorithms and its manifold regularized variants (graph sparse coding and Laplacian sparse coding), learn th…
Federated learning is a method of training a global model from decentralized data distributed across client devices. Here, model parameters are computed locally by each client device and exchanged with a central server, which aggregates the local models for a global view, without requiring sharing of training data. The…
Quantum Sparse Codingquant-ph
Quantum algorithm improves sparse vector recovery from noisy measurements.
problem Accurately recover sparse vectors from noisy linear measurements.
method Formulated as a QUBO task, solved using quantum technology.
result Quantum approach outperforms classical methods in sparse coding.
We present ErasureHead, a new approach for distributed gradient descent (GD) that mitigates system delays by employing approximate gradient coding. Gradient coded distributed GD uses redundancy to exactly recover the gradient at each iteration from a subset of compute nodes. ErasureHead instead uses approximate gradien…
Distributed gradient descent (DGD) is an efficient way of implementing gradient descent (GD), especially for large data sets, by dividing the computation tasks into smaller subtasks and assigning to different computing servers (CSs) to be executed in parallel. In standard parallel execution, per-iteration waiting time …
A new model for sequential memory using temporal predictive coding.
problem Forming accurate memory of sequential stimuli in the brain.
method Proposes a novel PC-based model called temporal predictive coding (tPC).
result Shows that tPC models can accurately memorize and retrieve sequential inputs.
The realized stochastic volatility (RSV) model that utilizes the realized volatility as additional information has been proposed to infer volatility of financial time series. We consider the Bayesian inference of the RSV model by the Hybrid Monte Carlo (HMC) algorithm. The HMC algorithm can be parallelized and thus per…
Long non-coding RNAs (lncRNAs) are a class of non-coding RNAs which play a significant role in several biological processes. RNA-seq based transcriptome sequencing has been extensively used for identification of lncRNAs. However, accurate identification of lncRNAs in RNA-seq datasets is crucial for exploring their char…
There is growing evidence regarding the importance of spike timing in neural information processing, with even a small number of spikes carrying information, but computational models lag significantly behind those for rate coding. Experimental evidence on neuronal behavior is consistent with the dynamical and state dep…
Neural machine translation models are used to automatically generate a document from given source code since this can be regarded as a machine translation task. Source code summarization is one of the components for automatic document generation, which generates a summary in natural language from given source code. Thi…
This work combines deep learning and sparse coding for CT image reconstruction.
problem Improving image quality in low-dose CT scans.
method Sparse signal representation using learned dictionaries, inspired by variational autoencoders and deep learning techniques.
result Regularization with learned dictionaries achieves competitive performance in CT reconstruction.
Paper explores understanding of neural source code embeddings.
problem Lack of understanding of contents and characteristics of code2vec embeddings.
method Small case study using code2vec embeddings to create binary SVM classifiers and compare performance with handcrafted features.
result Code2vec embeddings perform similarly to handcrafted features and have more evenly distributed information gains.
As the performance of computer systems stagnates due to the end of Moore's Law, there is a need for new models that can understand and optimize the execution of general purpose code. While there is a growing body of work on using Graph Neural Networks (GNNs) to learn representations of source code, these representation…
New model reveals balance crucial for robust neural coding.
problem Efficient neural coding in noisy, chaotic networks.
method Analytical model of balanced predictive coding with dissociated balance and weight disorder.
result Superclassical scaling in coding accuracy, independent of balance and weight disorder.