Paper introduces graph-based transforms for video compression.
problem Efficiently represent video signals for compression.
method Develops two techniques for designing graph-based transforms (GL-GBTs and EA-GBTs).
result Graph-based transforms outperform traditional KLT in video compression.
A new model designs molecules with desired properties.
problem Finding molecules with optimal chemical or biological properties.
method A latent prompt Transformer model with three components: latent vector, molecule generation, and property prediction.
result The model achieves state-of-the-art performance on molecule design tasks.
Transformer RL optimizes A/B testing for time series experiments.
problem Challenges in applying A/B testing to time series experiments, especially with limited history and strong assumptions.
method Transformer reinforcement learning approach that conditions allocation on full history and optimizes MSE without restrictive assumptions.
result Consistently outperforms existing designs in synthetic, simulator, and real-world data.
Data is said to follow the transform (or analysis) sparsity model if it becomes sparse when acted on by a linear operator called a sparsifying transform. Several algorithms have been designed to learn such a transform directly from data, and data-adaptive sparsifying transforms have demonstrated excellent performance i…
A new framework uses an Incremental Transformer to design geopolymer mixtures efficiently.
problem Designing geopolymer mixtures with limited data and physical constraints.
method Topology-aware surrogate framework guided by Incremental Transformer.
result The design space is redundant, with fewer effective mixture regimes.
ML predicts alloy properties considering chemistry, processing, and data transformations.
problem Designing and predicting alloy properties in high-dimensional design space.
method Physics-informed machine learning with engineered features from chemistry and heat treatment.
result ML models accurately predict alloy properties, including hysteresis in shape memory alloys.
Introduces Causal Energy Minimization to understand Transformer layers.
problem Empirical parameterization of Transformer blocks remains largely unexplored.
method Causal Energy Minimization framework that recasts Transformer layers as optimization steps on conditional energy functions.
result Identifies design space for Transformer layers including weight sharing and energy-based interpretations.
Study compares LSTM and Transformer models in financial time series prediction.
problem Comparing LSTM and Transformer models for financial time series prediction.
method Various LSTM-based and Transformer-based models compared on financial tasks; DLSTM and new Transformer architecture designed.
result Transformer-based models show limited advantage in absolute price sequence prediction, while LSTM-based models perform better on difference sequences.
Paper proposes linear transformers for efficient in-context learning without context length limitations.
problem Quadratic complexity of softmax transformers limits data processing speed.
method Investigates linear transformers under domain generalization, showing they learn mappings from context distributions to response functions.
result Linear transformers achieve in-context learning with a linear complexity in context length, offering a dimension-independent convergence rate.
The wavelet transform has seen success when incorporated into neural network architectures, such as in wavelet scattering networks. More recently, it has been shown that the dual-tree complex wavelet transform can provide better representations than the standard transform. With this in mind, we extend our previous meth…
A new framework designs experiments for better decision-making.
problem Suboptimal experimental designs for downstream decision-making.
method Amortized decision-aware Bayesian Experimental Design (BED) with Transformer Neural Decision Process (TNDP).
result TNDP effectively designs experiments and facilitates accurate decision-making.
NeuroSteiner uses neural networks to estimate wirelength more efficiently.
problem Minimizing wirelength in chip design.
method Neural model trained on synthesized nets to estimate WL.
result NeuroSteiner achieves 0.3% WL error at 60% faster than GeoSteiner.
Bayesian design improves experimental optimization.
problem Computational challenges limit BED practical use.
method Recent advances in BED have reduced computational burdens.
result Effective BED design is now feasible.
Using first principles from inference, we design a set of functionals for the purposes of \textit{ranking} joint probability distributions with respect to their correlations. Starting with a general functional, we impose its desired behaviour through the \textit{Principle of Constant Correlations} (PCC), which constrai…
New shape representation for airfoils improves design and manufacturing.
problem Designing and manufacturing airfoils efficiently and accurately.
method Combining physics-based and data-driven techniques on a Grassmannian manifold.
result Rich set of novel 2D airfoil deformations not previously captured.
Transforms offline greedy algorithms to online algorithms for combinatorial problems.
problem Online decision-making in time-varying combinatorial environments.
method General framework using Blackwell approachability and Bandit Blackwell approachability.
result Achieves O ( T ) O(\sqrt{T}) O ( T ) regret in full information setting and O ( T 2 / 3 ) O(T^{2/3}) O ( T 2/3 ) regret in bandit setting. In this paper, we explore creative image generation constrained by small data. To partially automate the creation of cartoon sketches consistent with a specific designer's style, where acquiring a very large original image set is impossible or cost prohibitive, we exploit domain specific knowledge for a huge reduction …
Computation of moments of transformed random variables is a problem appearing in many engineering applications. The current methods for moment transformation are mostly based on the classical quadrature rules which cannot account for the approximation errors. Our aim is to design a method for moment transformation for …
We introduce an application of the group lasso to design of experiments. Note that we are NOT trying to explain experimental design for the group lasso. Conversely, we explain how we can use the idea of the group lasso in experimental design, showing that the problem of constructing an optimal design matrix can be tran…
We propose a data-driven 3D shape design method that can learn a generative model from a corpus of existing designs, and use this model to produce a wide range of new designs. The approach learns an encoding of the samples in the training corpus using an unsupervised variational autoencoder-decoder architecture, withou…
Transformer-M learns molecular data in 2D or 3D formats.
problem Learning models for molecules are limited to specific data formats.
method Developed a Transformer-based model that can handle 2D and 3D molecular data.
result Transformer-M achieves strong performance on both 2D and 3D molecular tasks.
Energy Transformer integrates attention, energy models, and associative memory.
problem Lack of clear theoretical foundations in attention mechanisms and straightforward design of energy functions in energy-based models.
method Proposes Energy Transformer, a sequence of attention layers with a specifically engineered energy function.
result Obtained strong results on graph anomaly detection and classification tasks.
B-cos transformers explain Vision Transformers' decisions.
problem Lack of holistic explanations for transformer outputs.
method Formulate each component as dynamic linear, allowing a single linear transform for summarization.
result Bcos-ViTs are highly interpretable and competitive on ImageNet.
Layer normalization placement affects training stability and warm-up stage necessity.
problem Training instability and the necessity of a learning rate warm-up stage in Transformers.
method Theoretical analysis and mean field theory to prove gradient behavior at initialization.
result Removing the warm-up stage for Pre-LN Transformers can achieve comparable results with less time and tuning.
CRATE-MAE learns structured representations from unlabeled data.
problem Learning structured representations from unlabeled data.
method Structured Diffusion with White-Box Transformers.
result CRATE-MAE achieves highly promising performance on large-scale imagery datasets.
Paper improves Tm prediction of protein fragments using sparsity and probabilistic models.
problem Improving accuracy of melting temperature prediction for protein fragments.
method Promoting sparsity in pre-trained transformer models and adopting probabilistic frameworks.
result Mean absolute error of 0.23C for predicting melting temperature.
We introduce a new function-preserving transformation for efficient neural architecture search. This network transformation allows reusing previously trained networks and existing successful architectures that improves sample efficiency. We aim to address the limitation of current network transformation operations that…
SurvTRACE uses transformers to analyze survival times with competing events.
problem Analyzing survival times with multiple competing events and confounders.
method Transformer-based model for survival analysis, handling competing events and confounders.
result SurvTRACE outperforms existing methods in multi-event scenarios.
Tracr compiles programs into transformer models for interpretability.
problem Uncertainty in understanding transformer model outputs due to unknown learned programs.
method Tracr compiles human-readable programs into known structure transformer models.
result Known structure of Tracr-compiled models serves as ground-truth for interpretability.
Paper presents a Transformer model for automatic domain adaptation.
problem Challenges in selecting or designing domain adaptation algorithms.
method Transformer model approximates and selects domain adaptation algorithms.
result Transformers can approximate and automatically select domain adaptation algorithms.
ElasTST improves time-series forecasting across varying horizons.
problem Robust forecasting across different time horizons in varied industrial sectors.
method Elastic Time-Series Transformer (ElasTST) with non-autoregressive design, rotary position embedding, and multi-scale patching.
result ElasTST provides robust forecasts across varying horizons without retraining.
Efficient privacy-preserving machine learning framework using random transformations.
problem Slow training and inference speed in privacy-preserving machine learning systems.
method Random transformations like linear and permutation, combined with arithmetic sharing.
result High efficiency and low computation cost in private machine learning.
einspace expands NAS search space to include diverse neural architectures.
problem NAS results are often limited to existing structures; new designs are rare.
method einspace uses a probabilistic context-free grammar to create a versatile search space.
result einspace discovers novel and improved architectures, including convolutions and attention.
Method learns equivariances from data without custom architecture design.
problem Learning equivariances for tasks without manually designed architectures.
method Reparameterization to learn equivariant parameter sharing.
result Can learn equivariances for any finite group of transformations.
Transformers struggle with time series forecasting, especially in context.
problem Transformers' limitations in time series forecasting, particularly in in-context settings.
method Theoretical analysis of Transformers' limitations through In-Context Learning theory.
result Linear Self-Attention models cannot outperform classical linear models in in-context time series forecasting.
HGT models heterogeneous graphs with dedicated node and edge representations.
problem Infeasibility of existing GNNs for heterogeneous graphs.
method Node- and edge-type dependent parameters, relative temporal encoding, heterogeneous mini-batch graph sampling.
result HGT outperforms state-of-the-art GNNs by 9%--21% on various downstream tasks.
We propose an input design method for a general class of parametric probabilistic models, including nonlinear dynamical systems with process noise. The goal of the procedure is to select inputs such that the parameter posterior distribution concentrates about the true value of the parameters; however, exact computation…
Transformers mimic Bayesian reasoning in controlled settings, revealing geometric mechanisms.
problem Verifying if transformers perform Bayesian reasoning rigorously in natural data.
method Constructing Bayesian wind tunnels with known posteriors and proving memorization impossibility.
result Transformers achieve 10 − 3 10^{-3} 1 0 − 3 - 10 − 4 10^{-4} 1 0 − 4 bit accuracy in Bayesian posteriors, while MLPs fail. CogFormer trains a transformer to estimate parameters across various cognitive models.
problem Difficulty in fitting complex cognitive models and iterating over varying assumptions.
method Meta-amortized framework using a transformer to estimate parameters across multiple models.
result CogFormer accurately estimates parameters across different model families with minimal retraining.
We present a new construction for Poisson transforms between vector bundle valued differential forms on homogeneous parabolic geometries and the corresponding Riemannian symmetric space, which can be described in terms of finite dimensional representations of reductive Lie groups. In particular, we use these operators …
TAMO optimizes multiple objectives in-context using transformers.
problem Balancing competing objectives in expensive, black-box problems.
method Fully amortized, transformer-based policy for multi-objective optimization.
result Significant speedup in proposal time with improved Pareto quality.
A new transformer model accelerates training with optimization techniques.
problem Training deep neural networks efficiently and effectively.
method Interprets transformer layers as optimization steps, applying Nesterov acceleration.
result The new model outperforms existing models on benchmark datasets.
Improved hyperparameter optimization using simplified Transformer blocks.
problem Discovering optimal architectures in high-dimensional search spaces with limited exploration budgets.
method Simplified Transformer block for modeling hyper-parameter dependencies, actor-critic style algorithm, ensembling.
result Outperformed most algorithms on NAS-Bench-101 and Random Search in discovering more accurate model architectures.
Paper designs a lossy compression method for lossless prediction.
problem Ensuring high performance on predictive tasks with minimal data.
method Characterizes bit-rate requirements for invariant transformations, designs unsupervised objectives for neural compressors.
result Achieves substantial rate savings on ImageNet compared to JPEG without compromising classification performance.
This paper tackles causal representation learning with linear and general transformations.
problem Identify and recover latent causal variables and graphs under unknown transformations.
method Score-based algorithms that use gradients of log-density functions for identifiability and achievability.
result Two stochastic hard interventions per node are sufficient for identifiability of general transformations.
This study evaluates the robustness of transformation-based ensemble defense against evasion attacks.
problem Understanding the reasons behind the robustness improvement in transformation-based ensemble defense.
method Designing two adaptive attacks to evaluate transformation-based ensemble defense, conducting experiments to analyze robustness.
result The robustness improvement is mainly from irreversible transformations rather than the ensemble of models.
Transformer models outperform LSTM in financial forecasting with MADL loss.
problem Optimizing loss functions for Transformer models in financial forecasting.
method Empirical experiments with MADL loss function on equity and cryptocurrency assets.
result Transformer models significantly outperform LSTM models in financial forecasting.
Sparse transformer architecture improves accuracy and speed in generative modeling and inverse problems.
problem Improving accuracy and speed in generative modeling and inverse problems.
method Proposes a sparse transformer architecture using regularized Wasserstein proximal operator with L 1 L_1 L 1 prior. result Sparse transformer achieves higher accuracy and faster convergence than classical methods.