Researchers analyze neural process architectures and their representational capacities.
problem Understanding what functions can be represented by different neural process architectures.
method Analyzing four types of neural process architectures: CNPs, ANPs, TNPs, and their latent variants.
result Prove these architectures form a strict hierarchy and characterize their representational capabilities.
Enhances neural processes to learn from multiple related datasets.
problem Improving predictions from datasets with shared similarities.
method Developed the in-context in-context learning pseudo-token TNP (ICICL-TNP) to condition on both sets of datapoints and sets of datasets.
result Demonstrated the importance and effectiveness of in-context in-context learning.
CNNs encode data augmentation transformations, especially in early layers.
problem Whether neural network features encode data augmentation transformations.
method Systematic approach using pre-trained vision models to predict augmentation transformations.
result Neural network features encode data augmentation transformations, especially in early layers.
TNP-KR improves scalability of NPs with Transformer blocks and attention mechanisms.
problem Scalability bottleneck in NPs and GPs, especially for large datasets.
method Introduces TNP-KR with KRBlock, kernel-based attention, and two attention mechanisms.
result TNP-KR with DKA outperforms Performer and achieves state-of-the-art results.
NAST generalizes scattering transform for non-stationary time series analysis.
problem Analyzing non-stationary time series data.
method Neural activation of scattering transform with various activation functions and high pass filters.
result Central and non-central limit theorems for NAST of Gaussian processes.
Transformationally invariant processors constructed by transformed input vectors or operators have been suggested and applied to many applications. In this study, transformationally identical processing based on combining results of all sub-processes with corresponding transformations at one of the processing steps or …
BSA-TNP improves NP scalability and accuracy for spatiotemporal data.
problem Scalability and accuracy trade-off in Neural Processes.
method Introduces KRBlocks, group-invariant attention biases, and BSA for scalable spatiotemporal inference.
result BSA-TNP matches or exceeds accuracy of best models while training faster.
This paper proposes a new RV prediction model using neural distributional transformation and co-training.
problem Predicting skewed and fat-tailed realized volatility (RV) is challenging.
method The paper uses a neural distributional transformation and co-training to predict RV. It jointly trains the transformation and prediction model using a maximum-likelihood objective function.
result The proposed method significantly outperforms other methods on a dataset of 100 stocks.
Transformers struggle to approximate smooth functions, relying on piecewise constant approximations.
problem Understanding the expressivity of Transformers for function approximation.
method Theoretical analysis and experimental validation of Transformer's ability to approximate smooth functions.
result Transformers cannot reliably approximate smooth functions, relying on piecewise constant approximations.
This thesis explores GNNs, categorizing them into local and global approaches.
problem Understanding the convergence of global GNNs and connecting local and global approaches.
method Categorization of GNNs into local and global, study of Invariant Graph Networks, connecting local and global approaches, and using local MPNN for graph coarsening.
result Established a connection between local and global GNN approaches.
New translation equivariant neural processes improve spatio-temporal data modeling.
problem Improving posterior prediction maps for spatio-temporal data.
method Introduced translation equivariant transformers within neural processes.
result TE-TNPs outperform non-equivariant TNPs and other baselines.
New framework explains deep learning using signal processing techniques.
problem Lack of a rigorous mathematical theory explaining deep learning performance.
method Transform-domain sparse regularization, Radon transform, and approximation theory.
result Explains neural network properties and performance.
THP model captures complex dependencies in event sequences efficiently.
problem Inability of existing models to capture long-term dependencies in event sequences.
method Transformer Hawkes Process (THP) model using self-attention mechanism.
result THP outperforms existing models in likelihood and event prediction accuracy.
Paper improves neural ODEs for forecasting non-Markovian processes.
problem Forecasting irregularly observed time series with incomplete data.
method Path-dependent Neural Jump ODEs with signature transform.
result Path-dependent NJ-ODE outperforms original framework in non-Markovian data.
FourNet approximates financial transition densities using Fourier transforms.
problem Approximating transition densities in finance with high accuracy.
method FourNet is a novel FFNN with Gaussian activation, learning from characteristic functions.
result FourNet can approximate transition densities arbitrarily well with finite neurons.
CaTs use DAGs with transformers to enforce causal constraints, improving neural network robustness.
problem Neural networks lack inherent causal structure respect, leading to reliability issues.
method Introducing Causal Transformers (CaTs) that operate under predefined causal constraints specified by DAGs.
result CaTs improve robustness and interpretability of neural networks under causal constraints.
Improved weather forecasting with gridded pseudo-token TNPs.
problem Handling large-scale, unstructured spatio-temporal data in weather forecasting.
method Introducing gridded pseudo-token transformer neural processes (TNPs) with efficient attention mechanisms.
result Consistently outperforms baselines on various synthetic and real-world regression tasks involving large-scale data.
Efficient algorithm finds fast Transformer models.
problem Slow inference time of Transformer models.
method Decompose Transformer architecture into components, use sampling-based one-shot search.
result Achieved 10% to 30% speedup on pre-trained BERT and 70% on top of a previous state-of-the-art model.
Study Gaussian-process limits of neural networks using tensor programs.
problem Understanding the behavior of neural networks as they approach infinite width.
method Quantitative analysis through tensor programs and Wasserstein distance.
result Explicit finite-width error bounds, showing convergence to Gaussian-process limits.
HFNO enhances interpretability of turbulent flows through parallel wavenumber bin processing.
problem Opaque inner workings of Fourier Neural Operators (FNOs) hinder physical interpretability.
method Introduces HFNO, a novel FNO-based architecture that processes wavenumber bins in parallel, enhancing interpretability.
result HFNO decomposes turbulent flows across various scales, enabling increased interpretability and multiscale modeling.
The paper develops a computational method for efficient online filtering of diffusion processes.
problem Online filtering of discretely observed nonlinear diffusion processes.
method The approach involves Doob's h-transforms approximated by solving backward Kolmogorov equations using nonlinear Feynman-Kac formulas and neural networks. result The proposed method can be orders of magnitude more efficient than state-of-the-art particle filters.
This paper proposes a set of rules to revise various neural networks for 3D point cloud processing to rotation-equivariant quaternion neural networks (REQNNs). We find that when a neural network uses quaternion features under certain conditions, the network feature naturally has the rotation-equivariance property. Rota…
End-to-end neural network based approaches to audio modelling are generally outperformed by models trained on high-level data representations. In this paper we present preliminary work that shows the feasibility of training the first layers of a deep convolutional neural network (CNN) model to learn the commonly-used l…
GATES improves neural architecture search by modeling operations as information transformation.
problem Improving predictor-based neural architecture search efficiency.
method GATES models operations as information transformation, covering both node and edge cell search spaces.
result GATES boosts sample efficiency and improves predictor performance.
Mamba Hawkes Process improves modeling of event sequences with long-term dependencies.
problem Modeling mutual inhibition and nonlinearity in asynchronous event sequences.
method Introduces Mamba Hawkes Process using Mamba state space architecture.
result MHP outperforms existing models across various datasets.
New model improves field learning with improved equivariance.
problem Learning equivariant stochastic fields.
method Equivariant Gaussian processes and Steerable Conditional Neural Processes.
result SteerCNPs significantly improve performance in transfer learning tasks.
A new hierarchy quantifies agency in systems based on information processing.
problem Lack of a measurable, universal definition for agency in intelligent systems.
method Developed a bottom-up framework based on information processing hierarchy.
result Identified three orders of information processing (I, II, III) as necessary for agency.
Transformers can interpolate between arbitrary measures.
problem Understanding the expressive power of Transformers as measure-to-measure maps.
method Provided an explicit choice of parameters for a single Transformer to match N arbitrary input measures to N arbitrary target measures.
result A single Transformer can interpolate between arbitrary measures.
We introduce the implicit processes (IPs), a stochastic process that places implicitly defined multivariate distributions over any finite collections of random variables. IPs are therefore highly flexible implicit priors over functions, with examples including data simulators, Bayesian neural networks and non-linear tr…
Compact Recurrent Transformer (CRT) improves Transformer efficiency for long sequences.
problem Efficiently scaling Transformer architecture to long sequences with limited compute resources.
method Combines shallow Transformer models with recurrent neural networks and persistent memory.
result CRT achieves comparable or superior performance to full-length Transformers with shorter segments and reduced FLOPs.
WavPool improves deep neural networks with wavelet-based pooling.
problem Improving efficiency and performance of deep neural networks.
method Introducing WavPool, a wavelet-transform-based pooling layer.
result WavPool outperforms existing network architectures by 10% on CIFAR-10.
A framework for transformer attention layers derived from SVR.
problem Developing principled attention mechanisms for transformers.
method Mapping self-attention to SVR, deriving new attention types.
result Improved transformer performance and efficiency.
New Feedback Transformer architecture improves model performance by exposing past representations to future.
problem Limitations of Transformers in fully exploiting sequential input.
method Proposes Feedback Transformer exposing all past representations to future.
result Demonstrates improved performance with smaller, shallower models.
Graded Transformers embed algebraic structure in neural networks through graded transformations.
problem Efficiently modeling hierarchical and structured data in neural networks.
method Introduces Linearly Graded Transformer (LGT) and Exponentially Graded Transformer (EGT) with graded scaling operators.
result Establishes rigorous guarantees and improved efficiency for structured data.
Framework predicts Navier-Stokes solutions on 2D domains using graph neural networks.
problem Predicting stationary Navier-Stokes solutions in non-parametrized 2D geometries.
method Graph-based multi-fidelity learning framework combining reduced-order models, Transformers, and Mamba architectures.
result Mamba architecture reduces computational cost while maintaining performance.
Transformers' self-attention mechanism is mapped to a generalized Potts model.
problem Uncertainty in what type of data distribution self-attention can efficiently learn.
method Decouple word positions and embeddings, then show self-attention learns a generalized Potts model.
result Training self-attention is equivalent to solving the inverse Potts problem.
Transformers can learn optimal regression mixtures efficiently.
problem Limited adoption of tailored regression methods due to their model-specific nature.
method Constructed a generative process for a mixture of linear regressions and used transformers to learn optimal predictors.
result Transformers achieve low mean-squared error and make predictions close to the optimal procedure.
R2T hybrid model improves robust regression for asymmetric noise.
problem Least-squares regression fails with asymmetric structured noise.
method Transformer encoder, compression NN, fixed symbolic equation.
result Median regression MSE of 6e-6 to 3.5e-5 on synthetic data.
CTLNet predicts Shanghai Composite Index with improved accuracy.
problem Predicting Shanghai Composite Index for investors and researchers.
method Proposes CNN-Transformer-LSTM Networks (CTLNet) for multivariate time series forecasting.
result CTLNet outperforms state-of-the-art baselines in Shanghai Composite Index prediction.
MIMONets speed up neural network inference by processing multiple inputs in parallel.
problem Reducing computational cost in neural network inference for large datasets.
method Proposes MIMONets, which augment neural network architectures with variable binding mechanisms to handle multiple inputs in superposition.
result Achieves significant speedups (2-4x) with minimal accuracy loss, demonstrating adaptability across different architectures.
Inspired by the human visual perception system, hexagonal image processing in the context of machine learning deals with the development of image processing systems that combine the advantages of evolutionary motivated structures based on biological models. While conventional state-of-the-art image processing systems o…
NeuroMem-FHP framework estimates FHP parameters efficiently.
problem Estimating parameters of fractional Hawkes process (FHP) with long-range dependence.
method Developed LSTM and Transformer neural architectures to estimate FHP parameters directly from inter-arrival times.
result Transformer achieves highest estimation accuracy (MSE = 0.1634) compared to classical MLE (MSE = 2.8032).
Transformer models perform slower than convolutional networks in learning hierarchical language structures.
problem Understanding how neural networks learn hierarchical language structures.
method Theoretical scaling laws and empirical validation of neural network performance.
result Convolutional networks outperform transformers in learning hierarchical language structures.
Alternatives to recurrent neural networks, in particular, architectures based on attention or convolutions, have been gaining momentum for processing input sequences. In spite of their relevance, the computational properties of these alternatives have not yet been fully explored. We study the computational power of two…
Rough Transformers improve time series modeling with lower costs and better performance.
problem Inefficient modeling of irregularly sampled time series data.
method Signature patching for continuous-time representations, reducing computational costs.
result Rough Transformers outperform vanilla Transformers and Neural ODE models.
We applied pre-defined kernels also known as filters or masks developed for image processing to convolution neural network. Instead of letting neural networks find its own kernels, we used 41 different general-purpose kernels of blurring, edge detecting, sharpening, discrete cosine transformation, etc. for the first la…
A method uses image processing and deep learning for financial market state prediction.
problem Low signal-to-noise ratio in financial time series data.
method Wavelet transform for denoising, convolutional neural network for pattern extraction.
result Competitive prediction accuracy of market states 'Up' and 'Down' on S&P 500 data.
Convolutional neural networks (CNNs) have achieved state-of-the-art results on many visual recognition tasks. However, current CNN models still exhibit a poor ability to be invariant to spatial transformations of images. Intuitively, with sufficient layers and parameters, hierarchical combinations of convolution (matri…