A new framework learns system design using neural features in function space.
problem Learning system design with neural feature extractors.
method Introduces feature geometry in function space, nesting technique for optimal feature approximation.
result Optimal features found from data samples using off-the-shelf architectures and optimizers.
Study compares LSTM and ANN architectures for forex prediction, finding ANN more efficient.
problem Forex time series prediction efficiency and resource usage.
method Comparison of Long Short-Term Memory (LSTM) and specialized ANN architectures.
result Specialized ANN architecture performs better with fewer resources and faster execution.
We provide a comprehensive investigation of different custom and off-the-shelf architectures as well as different approaches to generating feature vectors for offensive language detection. We also show that these approaches work well on small and noisy datasets such as on the Offensive Language Identification Dataset (…
EASIER-net uses sparse networks to improve prediction accuracy for high-dimensional data.
problem Limited use of neural networks in high-dimensional data with small samples.
method Ensemble by Averaging Sparse-Input Hierarchical networks (EASIER-net) with small modifications to neural network architecture and training procedure.
result EASIER-net achieves higher prediction accuracy than off-the-shelf methods on average.
We reformulate the option framework as two parallel augmented MDPs. Under this novel formulation, all policy optimization algorithms can be used off the shelf to learn intra-option policies, option termination conditions, and a master policy over options. We apply an actor-critic algorithm on each augmented MDP, yieldi…
This study evaluates different normalizing flow architectures for MCMC.
problem Lack of systematic comparison of normalizing flow architectures in MCMC.
method Extensive evaluation of various normalizing flow architectures on different MCMC methods and target distributions.
result Contractive residual flows are the best general-purpose models for MCMC.
New method trains any neural network as a generative model.
problem Constrained design of normalizing flows due to analytical invertibility.
method Efficient gradient estimator for non-analytically invertible networks.
result Any dimension-preserving neural network can be used as a generative model.
WideBNet learns inverse scattering from wide-band data efficiently and stably.
problem Learning the inverse scattering map from wide-band scattering data.
method Combines butterfly factorization, FFT, and deep learning.
result WideBNet requires fewer training points and has stable training dynamics.
This work uses image generation models to find vision model bugs.
problem Automatically discovering failures in vision models.
method Conditional text-to-image generation and captioning models.
result Demonstrated utility of large-scale generative models to find vision model bugs.
In this paper, we introduce multi-task learning (MTL) to data harmonization (DH); where we aim to harmonize images across different acquisition platforms and sites. This allows us to integrate information from multiple acquisitions and improve the predictive performance and learning efficiency of the harmonization mode…
JD.com uses a new CNN model to improve ad click prediction.
problem Improving CTR prediction for ads with visual content.
method Proposes Category-specific CNN (CSCNN) to incorporate category knowledge early in the feature extraction process.
result CSCNN outperforms existing methods in CTR prediction.
Labels distilled from images improve model training efficiency and flexibility.
problem Creating synthetic labels for a small set of real images to train models effectively.
method Introduce a more robust and flexible meta-learning algorithm for distillation and an effective first-order strategy based on convex optimization layers.
result Label distillation leads to improved results and greater flexibility in neural architectures.
We present a generative framework for generalized zero-shot learning where the training and test classes are not necessarily disjoint. Built upon a variational autoencoder based architecture, consisting of a probabilistic encoder and a probabilistic conditional decoder, our model can generate novel exemplars from seen/…
We formalize the problem of trading-off DNN training time and memory requirements as the tensor rematerialization optimization problem, a generalization of prior checkpointing strategies. We introduce Checkmate, a system that solves for optimal rematerialization schedules in reasonable times (under an hour) using off-t…
Prior domain knowledge can greatly help to learn generative models. However, it is often too costly to hard-code prior knowledge as a specific model architecture, so we often have to use general-purpose models. In this paper, we propose a method to incorporate prior knowledge of feature relations into the learning of g…
New method uses off-the-shelf classifiers to improve diffusion generation without extra training.
problem Improving sample quality and controllability in conditional diffusion generation.
method Leveraging off-the-shelf classifiers in a training-free fashion with calibration and pre-conditioning techniques.
result Significant performance improvements (up to 20%) over existing guidance schemes.
Generative adversarial networks benefit from optimal input dimension and adaptive generator architecture.
problem Minimizing generalization error in GANs through optimal input dimension.
method Introducing generalized GANs (G-GANs) with group penalty and architecture penalty for adaptive dimensionality reduction and network architecture identification.
result G-GANs achieve superior performance with 40%+ improvements in maximum mean discrepancy or Frechet inception distance compared to off-the-shelf methods.
SymDiff uses stochastic symmetrisation for equivariant diffusion models.
problem Constructing equivariant diffusion models for data augmentation.
method Stochastic symmetrisation for lightweight, efficient, and easy-to-implement equivariance.
result SymDiff achieves significant empirical benefit for E(3)-equivariant molecular generation. Rapid improvements in machine learning over the past decade are beginning to have far-reaching effects. For communications, engineers with limited domain expertise can now use off-the-shelf learning packages to design high-performance systems based on simulations. Prior to the current revolution in machine learning, th…
BREEDS benchmarks assess model robustness to subpopulation shifts.
problem Measuring model robustness to novel subpopulation shifts.
method Controlled synthesis of realistic distribution shifts using class structure.
result Validated model sensitivity and effectiveness of robustness interventions.
State-of-the-art speaker diarization systems utilize knowledge from external data, in the form of a pre-trained distance metric, to effectively determine relative speaker identities to unseen data. However, much of recent focus has been on choosing the appropriate feature extractor, ranging from pre-trained i−vectors…
A new method recovers rewards from behavior policies using classification and regression.
problem Recovering meaningful rewards from observed behavior in reinforcement learning.
method GenPQR, a modular procedure that estimates behavior policy, evaluates soft Q-function, and recovers normalized reward using classification and regression.
result GenPQR matches or improves reward recovery compared to DeepPQR, while being simpler and more modular.
BERT-XML automates ICD coding from EHR notes using BERT pretraining.
problem Manual ICD coding is time-consuming and expensive in hospitals.
method Trained a BERT model on EHR notes, adapting it for multi-label attention.
result First large-scale ICD-10 classification model using millions of EHR notes.
Modern neural networks are highly overparameterized, with capacity to substantially overfit to training data. Nevertheless, these networks often generalize well in practice. It has also been observed that trained networks can often be "compressed" to much smaller representations. The purpose of this paper is to connect…
New insights into attention mechanisms reveal dramatic trade-offs between rank and heads.
problem Dramatic trade-offs between rank and number of heads in attention mechanisms.
method Presented a simple target function and proved theoretical limits.
result Full-rank attention is necessary for long contexts, while low-rank is sufficient for short ones.
SS-GEN simulates rare events in heavy and light-tailed data.
problem Estimating probabilities of extreme events in multivariate data.
method Self-Similar Generative Estimation (SS-GEN) decomposes tail distribution into radial and angular components.
result SS-GEN generates representative extreme scenarios and estimates rare-event probabilities beyond observed data.
We characterize a prevalent weakness of deep neural networks (DNNs)---overthinking---which occurs when a DNN can reach correct predictions before its final layer. Overthinking is computationally wasteful, and it can also be destructive when, by the final layer, a correct prediction changes into a misclassification. Und…
Dynamic residual adapters improve performance across multiple latent domains without domain labels.
problem Overfitting to large domains and ignoring smaller ones in multi-domain learning.
method Dynamic residual adapters and augmentation strategies inspired by style transfer.
result Dynamic residual adapters significantly outperform standard models on multiple latent domains.
Defines metrics to compare neural network representations.
problem Comparing neural network representations across different architectures and tasks.
method Developed a family of metric spaces and modified existing measures to quantify representational dissimilarity.
result Identified relationships between neural representations and anatomical features.
New L1-Coverage objective simplifies exploration in reinforcement learning.
problem Challenges in exploration for high-dimensional domains.
method Introduces L1-Coverage objective to enable efficient exploration and planning. result First computationally efficient algorithms for online reinforcement learning with low coverability.
Benchopt automates machine learning benchmarking across languages and hardware.
problem Limited transparency and tedious re-implementation work in machine learning validation.
method A collaborative framework for automating, reproducing, and publishing optimization benchmarks.
result Demonstrates practical findings that highlight the importance of details in machine learning validation.
Improves ASR accuracy for domain mismatch using machine translation.
problem Domain mismatch in ASR systems leads to suboptimal results.
method Machine translation to map out-of-domain ASR errors to in-domain terms.
result 7% absolute improvement in word error rate, 4 point BLEU score improvement.
CoCoPIE shows AI can run on regular devices without special hardware.
problem Real-time AI on mobile devices without special hardware.
method Compression-compiler co-design using CoCoPIE framework.
result Real-time AI applications on off-the-shelf mobile devices.
PAVI speeds up Bayesian inference for large datasets.
problem Challenges in Bayesian inference for large population studies.
method Plate-amortized Variational Inference (PAVI) that shares parameterization across i.i.d. variables.
result Significant speedup in training variational distributions, orders of magnitude faster.
We are in the middle of a remarkable rise in the use and capability of artificial intelligence. Much of this growth has been fueled by the success of deep learning architectures: models that map from observables to outputs via multiple layers of latent representations. These deep learning algorithms are effective tools…
This paper examines data transfer methods to improve off-the-shelf Transformer models for retrosynthesis.
problem Improving prediction models for retrosynthesis using small datasets.
method Systematic examination of data transfer approaches on end-to-end generative models.
result Pre-training plus fine-tuning boosts the accuracy of a Transformer baseline, achieving state-of-the-art results.
As an effective data preprocessing step, feature selection has shown its effectiveness to prepare high-dimensional data for many machine learning tasks. The proliferation of high di-mension and huge volume big data, however, has brought major challenges, e.g. computation complexity and stability on noisy data, upon exi…
Novel approach using Lagrangian Decomposition for neural network verification.
problem Computing tight bounds on neural network outputs.
method Lagrangian Decomposition, efficient supergradient ascent, proximal algorithm.
result Proves tighter bounds than previous dual algorithms.
This study investigates how much knowledge from natural images can be transferred to pathology images.
problem Quantifying how much knowledge from natural images can be transferred to pathology images.
method Proposes a framework to quantify knowledge gain by a particular layer, conducts empirical investigation in pathology image centered transfer learning.
result Early layers of deep models can transfer knowledge to pathology image classification tasks.
Wasserstein distance-based distributionally robust optimization (DRO) has received much attention lately due to its ability to provide a robustness interpretation of various learning models. Moreover, many of the DRO problems that arise in the learning context admits exact convex reformulations and hence can be tackled…
This paper proposes to learn hierarchical compositional AND-OR model for interpretable image synthesis by sparsifying the generator network. The proposed method adopts the scene-objects-parts-subparts-primitives hierarchy in image representation. A scene has different types (i.e., OR) each of which consists of a number…
Since its introduction, unsupervised representation learning has attracted a lot of attention from the research community, as it is demonstrated to be highly effective and easy-to-apply in tasks such as dimension reduction, clustering, visualization, information retrieval, and semi-supervised learning. In this work, we…
Expands causal clustering framework with hierarchical and density-based methods.
problem Identifying heterogeneous treatment effects in unknown subgroup structure.
method Integrates hierarchical and density-based clustering algorithms into causal k-means clustering.
result Plug-in estimators for causal clustering are simple and readily implementable.
Optimizes matching in weighted graphs with semi-bandit sampling.
problem Finding optimal pairings in weighted graphs with sequential sampling.
method Leverages rank-1 assumption on adjacency matrix to reduce sample complexity and regret.
result Achieves linear dependency in the number of vertices for sample complexity and regret.
Investigates numerical issues in GP interpolation parameter estimation.
problem Numerical issues in maximum likelihood parameter estimation for Gaussian process interpolation.
method Investigates and proposes strategies to improve open-source software implementations.
result Improves reliability and reproducibility of studies relying on GP implementations.
Estimates sparse Gaussian graphical models using discrete optimization.
problem Learning a sparse graph from Gaussian graphical models.
method Proposes GraphL0BnB, an ℓ0-penalized MIP solved with a custom BnB framework. result Significant runtime and statistical performance improvements over existing methods.
We provide a comprehensive review of classical algorithms for compressive sensing of images, focused on Total variation methods, with a view to application in LiDAR systems. Our primary focus is providing a full review for beginners in the field, as well as simulating the kind of noise found in real LiDAR systems. To t…
Deep neural networks trained using a softmax layer at the top and the cross-entropy loss are ubiquitous tools for image classification. Yet, this does not naturally enforce intra-class similarity nor inter-class margin of the learned deep representations. To simultaneously achieve these two goals, different solutions h…