Spectral regularization simplifies sequence models by focusing on grammatical simplicity.
problem Sequence modeling challenges in learning tasks.
method Introduces spectral regularization based on Hankel matrices and trace norm, addressing bi-infinite matrices with an unbiased estimator.
result Demonstrates spectral regularization's potential benefits on Tomita grammars.
This work verifies recurrent neural networks using DFA and shows some are robust to adversarial samples.
problem Verifying recurrent neural networks for adversarial robustness.
method Metric for string distance, DFA for oracle, Tomita grammars for testing.
result Some recurrent networks are robust to adversarial samples.
New grammar model learns sentence structure with latent variables.
problem Grammar induction for sentences with complex dependencies.
method Compound probabilistic context-free grammar with latent variables, variational inference.
result Effective unsupervised parsing compared to state-of-the-art methods.
Probabilistic grammars improve equation discovery from data.
problem Discovering scientific laws from data using equations.
method Proposed probabilistic context-free grammars to encode soft constraints and a Monte-Carlo algorithm.
result Probabilistic grammars lead to more efficient equation discovery.
Constructs a lift of fusion for spinors on the circle using Tomita-Takesaki theory.
problem Fusion of operators on spinors on the circle.
method Uses Tomita-Takesaki theory for Clifford-von Neumann algebras.
result Lifts fusion to a central extension of implementers on a Fock space.
AG-RL uses action grammars to improve reinforcement learning efficiency.
problem Improving sample efficiency in reinforcement learning.
method Integrates action grammars into reinforcement learning algorithms to enhance performance.
result Significant improvement in performance across multiple Atari games.
Unified recurrent networks reveal differences in complexity levels of grammars.
problem Understanding the complexity and behavior of recurrent networks.
method Connecting recurrent networks with deterministic finite automata and formal grammars.
result Unified recurrent networks improve performance and match grammars from different complexity levels.
Unsupervised RNNGs perform similarly to supervised ones in language modeling and grammar induction.
problem Training RNNGs requires annotated data, which is costly.
method Amortized variational inference with a neural CRF parser.
result Unsupervised RNNGs achieve comparable performance to supervised ones.
AlphaCFG discovers alpha factors using grammar-guided search.
problem Discovering formulaic alpha factors in finance.
method AlphaCFG uses a grammar-based framework to define and discover alpha factors with syntactic and semantic constraints.
result AlphaCFG outperforms state-of-the-art methods in trading profitability and efficiency.
Future predictions on sequence data (e.g., videos or audios) require the algorithms to capture non-Markovian and compositional properties of high-level semantics. Context-free grammars are natural choices to capture such properties, but traditional grammar parsers (e.g., Earley parser) only take symbolic sentences as i…
We propose a new statistical model for computational linguistics. Rather than trying to estimate directly the probability distribution of a random sentence of the language, we define a Markov chain on finite sets of sentences with many finite recurrent communicating classes and define our language model as the invarian…
Codebook for Institutional Grammar 2.0 simplifies policy encoding.
problem Facilitating consistent policy encoding for diverse analytical needs.
method Revised Institutional Grammar with multiple levels of expressiveness.
result Enhanced flexibility and specificity in policy encoding.
Efficient molecular optimization using probabilistic hypergraph grammars and reinforcement learning.
problem Optimizing molecular structures for better performance in various benchmarks.
method Inferred a hypergraph replacement grammar from ChEMBL, used conditional priors for policy model, and applied a modified policy gradient algorithm.
result The approach results in a molecular distribution closer to the training set than using equal or unconditional priors.
The task of translating between programming languages differs from the challenge of translating natural languages in that programming languages are designed with a far more rigid set of structural and grammatical rules. Previous work has used a tree-to-tree encoder/decoder model to take advantage of the inherent tree s…
Proposes a method to train neural networks directly on compressed text data.
problem Training neural networks on compressed text data without decompression.
method Introduces composer modules to encode symbols from grammar compression rules into vector representations.
result Demonstrates that the proposed method can achieve both memory and computational efficiency while maintaining moderate performance.
Improved AutoML performance with a pipeline grammar and pre-trained model.
problem Automatic machine learning by pipeline synthesis.
method Model-based reinforcement learning and pipeline grammar.
result Improved performance on AutoML benchmark datasets.
Net2Vis automates CNN visualization for publications.
problem Lack of consistent visual representations in deep learning papers.
method Proposes a visual grammar and automated system for generating publication-ready CNN visualizations.
result Reduces time and ambiguity in generating network visualizations.
Automates molecule design with simpler SMILES generation and reinforcement learning.
problem Designing molecules with specific chemical properties.
method Combines context-free grammar for SMILES strings and reinforcement learning with a Transformer model.
result Significantly reduces model steps per atom and beats previous baselines.
Ensemble method detects time series anomalies without preselecting parameter values.
problem Anomaly detection requires known anomaly length, limiting practicality.
method Ensemble grammar induction for variable-length anomalies.
result Ensemble approach outperforms existing methods with different parameter selections.
We introduce a C*-algebra A(x,Q) attached to the cluster x and a quiver Q. If Q(T) is the quiver coming from a triangulation T of the Riemann surface S with a finite number of cusps, we prove that the primitive spectrum of A(x,Q(T)) times R is homeomorphic to a generic subset of the Teichmueller space of surface S. We …
A new CNN-based code generator outperforms RNNs by 5%.
problem Capturing long sequences in code generation using RNNs.
method Grammar-based structural CNN with tree-based and pre-order convolution modules.
result Significantly outperforms previous state-of-the-art methods by 5 percentage points.
Method extracts knowledge from LSTM for sequence validation.
problem Validating sequences generated by unknown automata.
method Clustering hidden states to build a corresponding automaton.
result Automaton accurately predicts sequence validity.
TES-AE uses tree grammars to speed up autoencoding for tree data.
problem Challenges in autoencoding tree data due to its non-vectorial and discrete nature.
method TES-AE combines reservoir computing with tree grammars for faster training.
result TES-AE outperforms D-VAE in speed and accuracy for tree data.
Molecular optimization aims to discover novel molecules with desirable properties. Two fundamental challenges are: (i) it is not trivial to generate valid molecules in a controllable way due to hard chemical constraints such as the valency conditions, and (ii) it is often costly to evaluate a property of a novel molecu…
Proposes a framework for compositional generalization in language models.
problem Lack of compositional generalization in neural networks compared to humans.
method Introduces Generalized Grammar Rules (GGRs) for transduction tasks, formalizing symmetry-based constraints.
result Framework enables models to generalize compositionally, similar to human learning.
Tree-Transformer improves grammar correction in code and natural language.
problem Grammar correction in tree-structured data.
method Tree-Transformer neural network architecture for tree-structured data.
result Significant improvement in grammar correction accuracy.
Loops in surfaces and chord diagrams are studied with graph factorizations and grammars.
problem Understanding loops in surfaces and their properties.
method Factorization of filoops into spheric and toric sums, and grammars generating chordiagraphs.
result Minimal genus of filoops and stability properties under factorizations.
Traditional Linear Genetic Programming (LGP) algorithms are based only on the selection mechanism to guide the search. Genetic operators combine or mutate random portions of the individuals, without knowing if the result will lead to a fitter individual. Probabilistic Model Building Genetic Programming (PMB-GP) methods…
Paper closes neural-symbolic learning loop with grammar model and back-search algorithm.
problem Slow convergence in neural-symbolic learning due to error propagation issues.
method Introduces grammar model as symbolic prior and back-search algorithm for efficient error propagation.
result Significantly outperforms RL methods in performance, converging speed, and data efficiency.
A framework infers causal direction from symbolic sequences using compression measures.
problem Inferring causal direction from two observed discrete symbolic sequences.
method Lossless compressors for inferring context-free grammars (CFGs) and quantifying compression extent.
result Grammar inferred from one sequence better compresses the other sequence, indicating causal direction.
Interdisciplinary comparison of sequence modeling methods for next-element prediction.
problem Comparing sequence modeling methods across different fields.
method Experimental evaluation of four real-life sequence datasets using machine learning, process mining, and grammar inference techniques.
result Machine learning techniques outperform interpretability-focused methods in next-element prediction accuracy.
The recent proliferation of richly structured probabilistic models raises the question of how to automatically determine an appropriate model for a dataset. We investigate this question for a space of matrix decomposition models which can express a variety of widely used models from unsupervised learning. To enable mod…
A new framework generates large hierarchical search spaces for neural architectures.
problem Discovering neural architectures from simple blocks is hard.
method Context-free grammars for a unified, scalable search space.
result Efficiently searches over complete architectures, outperforming existing methods.
Describes MLC search spaces in MEKA and WEKA software.
problem Understanding MLC algorithms and their transformations into SLC problems.
method Overviewed 26 MLC algorithms and 28 SLC algorithms, proposed a context-free grammar.
result Formal description of MLC search spaces and their transformations.
Cone structures in quantum field theory linked to information geometry.
problem Understanding geometric structures in quantum field theory.
method Analyzing invariant cones under modular automorphism groups and their relation to Wishart laws.
result Explicit connection between CAH cones and Wishart laws.
Unsupervised algorithm parses CSG images into CFG without pretraining.
problem Sparse reward problem in unsupervised program synthesis for images.
method Grammar-encoded tree LSTM, entropy regularization, sampling without replacement.
result Recover meaningful programs in large search spaces (up to 3.8imes1028). We introduce Markov substitute processes, a new model at the crossroad of statistics and formal grammars, and prove its main property : Markov substitute processes with a given support form an exponential family.
BOSS optimizes string inputs using string kernels and genetic algorithms.
problem Optimizing string inputs with constraints.
method Bayesian optimization over string kernels and genetic algorithms.
result Significantly improved optimization across various string constraints.
Unified model improves coherence tasks, especially local contexts.
problem Existing neural coherence models struggle with local context tasks.
method Unified neural framework integrating grammar, relations, and patterns.
result Unified model outperforms existing models significantly.
Study shows mutual information can reward structure learning agents without expert systems.
problem Designing rewards for structure learning agents in natural language environments.
method Revisited Information Theory of unsupervised induction of phrase-structure grammars, using random sets of linguistic samples.
result Empirical evidence that simulated semantic structures can be distinguished from random ones by mutual information among their constituents.
Deep generative models have been wildly successful at learning coherent latent representations for continuous data such as video and audio. However, generative modeling of discrete data such as arithmetic expressions and molecular structures still poses significant challenges. Crucially, state-of-the-art methods often …
Diffusion models learn hierarchical composition rules from data.
problem How many samples do generative models need to learn hierarchical composition rules?
method Theoretical and empirical investigation of diffusion models on probabilistic context-free grammars.
result Diffusion models learn hierarchical composition rules with sample complexity scaling polynomially with context size.
Unified theory for neural scaling laws in hierarchically compositional data.
problem Understanding neural scaling laws in hierarchically compositional data.
method Probabilistic context-free grammars and power-law distributed production rules.
result Unified learning curve behavior for classification and next-token prediction tasks.
In this contribution we describe an approach to evolve composite covariance functions for Gaussian processes using genetic programming. A critical aspect of Gaussian processes and similar kernel-based models such as SVM is, that the covariance function should be adapted to the modeled data. Frequently, the squared expo…
Capsule networks improve at detecting changes in compositionality with routing.
problem Capsule networks struggle with detecting changes in compositionality.
method Introduced a loss function based on routing entropy to improve compositionality.
result Capsule networks with the new loss function better detect changes in compositionality.
Deep learning is currently the subject of intensive study. However, fundamental concepts such as representations are not formally defined -- researchers "know them when they see them" -- and there is no common language for describing and analyzing algorithms. This essay proposes an abstract framework that identifies th…
Paper details Hilbert-curve for high-performance data mining.
problem Efficiently mapping multi-dimensional data to one dimension.
method Defines Hilbert-curve using finite automaton and context-free grammar.
result Cache-oblivious algorithms for matrix operations and clustering.
Early stopping improves generalization in overparameterized diffusion models.
problem Understanding and optimizing generalization in overparameterized diffusion models.
method Revisiting diffusion models, showing generalization occurs before memorization, and developing a phase diagram.
result Generalization time scales with dataset size, supporting early-stopping criteria.