Proposes ReDT for interpretable, compressed, and robust decision trees.
problem Improving interpretability and performance of decision trees.
method Knowledge distillation with soft labels and multiple cross-validation.
result ReDT achieves fewer nodes than classical decision trees while maintaining good performance and interpretability.
Bayesian technique compresses RNNs by 100x without tuning.
problem Large parameter size in RNNs, especially embeddings.
method Bayesian sparsification for RNNs and vocabulary filtering.
result RNNs can be compressed dozens or hundreds of times.
Convolutional neural networks (CNNs) achieve state-of-the-art performance in a wide variety of tasks in computer vision. However, interpreting CNNs still remains a challenge. This is mainly due to the large number of parameters in these networks. Here, we investigate the role of compression and particularly pruning fil…
Dirichlet pruning compresses neural networks by removing unimportant units.
problem Compressing large neural network models without sacrificing performance.
method Assigns Dirichlet distribution over network layers' units and uses variational inference to estimate parameters.
result Achieves state-of-the-art compression performance on larger architectures like VGG and ResNet.
Model compression improves dynamic forecasting ensembles while reducing computational costs.
problem High computational costs and lack of transparency in dynamic forecasting ensembles.
method Model compression applied to dynamic forecasting ensembles of various types of models.
result Compressed models achieve comparable predictive performance and significant computational savings.
Proposes SHORE model for efficient MOR with sparsity and scalability.
problem Challenges of interpretability and scalability in MOR with high-dimensional outputs.
method Incorporates sparsity requirements and a two-stage optimization framework for efficient compression.
result Theoretical and empirical validation of the proposed framework's efficiency and accuracy.
Over the years, ensemble methods have become a staple of machine learning. Similarly, generalized linear models (GLMs) have become very popular for a wide variety of statistical inference tasks. The former have been shown to enhance out- of-sample predictive power and the latter possess easy interpretability. Recently,…
Generative model learns wireless channel distributions efficiently.
problem Learning precise wireless channel distributions for optimal communication.
method Physics-informed sparse Bayesian generative modeling (SBGM) with compressed data.
result Model learns channel parameters from compressed AP observations, is physically interpretable, and generalizes across different systems.
Image compression techniques reveal network structure for shipping box optimization.
problem Designing and planning shipping box networks between nodes.
method Image transformations and Bayesian reinforcement learning.
result Learned network structure can recommend future connectivity.
Scattering representations simplify SBI for images without extra compression.
problem Efficiently performing simulation-based inference on images with limited data.
method Use scattering representations for compression and learning, combined with spatial averaging and expressive density estimators.
result Scattering representations provide more information than traditional methods, without requiring additional simulations.
Bayesian method sparsifies gated RNNs, improving speed and interpretability.
problem Sparsifying neural networks to reduce complexity and improve performance.
method Bayesian approach to sparsify weights, neurons, and gates in LSTM architectures.
result Sparsified gated RNNs speed up forward pass and improve compression.
Model compression reduces generalization error and increases empirical risk, potentially improving population risk.
problem Improving population risk with model compression.
method Information-theoretic analysis and rate distortion theory.
result Model compression can improve population risk if the decrease in generalization error exceeds the increase in empirical risk.
LLMs compress financial texts, but distort decision-making.
problem LLMs compress financial texts, altering decision-making.
method Analyzed two diagnostic patterns: decontextualization and model dependency. Proposed Agentic Context Compression.
result LLM-compressed financial texts alter decision-making.
We address the problem of Compressed Sensing (CS) with side information. Namely, when reconstructing a target CS signal, we assume access to a similar signal. This additional knowledge, the side information, is integrated into CS via L1-L1 and L1-L2 minimization. We then provide lower bounds on the number of measuremen…
Compressed LLM embeddings improve noisy regression tasks without overfitting.
problem Noisy regression tasks with high signal-to-noise ratios.
method Comparison of embedding compression techniques using autoencoder hidden representations.
result Compression improves performance on noisy tasks like financial return prediction.
New method compresses deep learning layers using tensor decomposition.
problem Reduction of computation cost and interpretability for tensor data.
method CP-decomposition to compress convolutional layers in deep learning.
result Reduces model complexity and maintains prediction performance.
C2G-Net improves image classification of similar objects like cells.
problem Classifying images with many similar objects efficiently and interpretably.
method Combines image compression and a CNN with reduced parameters.
result C2G-Net achieves similar accuracy to conventional CNNs but with reduced training time and improved interpretability.
New theory controls compression change probability without prior knowledge.
problem Controlling compression change probability without prior knowledge.
method New theory on compression function and statistical risk.
result Cardinality of compressed set is a consistent estimator of probability of change of compression.
A new method compresses conditional distributions of labelled data.
problem No existing method directly compresses the conditional distribution of labelled data.
method Introduce Average Maximum Conditional Mean Discrepancy (AMCMD), derive a closed form estimator, and extend Kernel Herding (KH) to Average Conditional Kernel Herding (ACKH).
result Directly compressing conditional distributions outperforms joint distribution compression and greedy selection.
This work considers an estimation task in compressive sensing, where the goal is to estimate an unknown signal from compressive measurements that are corrupted by additive pre-measurement noise (interference, or clutter) as well as post-measurement noise, in the specific setting where some (perhaps limited) prior knowl…
SANs use sparse activation functions to compress data representations.
problem Learning meaningful features without considering compression.
method Introduce φ metric, define activation functions, and present SANs.
result SANs achieve small description length and interpretable kernels.
CIB compresses variables causally, preserving key causal interactions.
problem Constructing causal variable abstractions in complex systems.
method Causal Information Bottleneck (CIB) method, extending IB to include causal structures.
result CIB produces causally interpretable abstractions that accurately capture causal relations.
We simplify complex regression coefficients using linearization and feature comparison.
problem Interpreting high-dimensional regression coefficients from nonlinear responses.
method Developed a linearization method to derive feature coefficients and compare them with regression coefficients.
result Shows how regression coefficients relate to linearized feature coefficients and how they change under regularization.
APD method decomposes neural network parameters into simple, faithful components.
problem Understanding the internal mechanisms learned by neural networks.
method Attribution-based Parameter Decomposition (APD) method.
result Demonstrated effectiveness in recovering features, separating computations, and identifying representations.
A novel method compresses point cloud attributes by folding them onto a 2D grid.
problem Efficiently compressing point cloud attributes for storage and transmission.
method Interpreting point clouds as 2D manifolds, folding onto a grid, and mapping attributes to the grid using optimized methods.
result The proposed folding-based approach achieves performance comparable to state-of-the-art codecs.
A novel pruning method finds relevant units in CNNs for efficient compression.
problem Reduction of computation and storage costs in deep neural networks.
method Pruning by explaining, using relevance scores from explainable AI.
result The method efficiently compresses CNN models without sacrificing performance.
NICE framework explains CNN predictions and compresses images.
problem Interpreting deep neural networks and efficient image compression.
method End-to-end Neural Image Compression and Explanation (NICE) framework.
result NICE achieves high compression rates while maintaining classification accuracy.
New measure LMN explains neural network grokking.
problem Delayed generalization after memorization in neural networks.
method Defined LMN to measure network complexity, showing LMN correlates with test losses linearly.
result LMN reveals intriguing XOR network behavior and is a promising complexity measure.
Paper proposes a hybrid model-based and data-driven approach for one-bit compressive autoencoding.
problem Designing efficient one-bit compressive autoencoding models for complex systems.
method Hybrid model-based and data-driven methodology for one-bit sparse signal recovery.
result Significant improvement in one-bit compressive autoencoding compared to state-of-the-art algorithms.
We propose tensorial neural networks (TNNs), a generalization of existing neural networks by extending tensor operations on low order operands to those on high order ones. The problem of parameter learning is challenging, as it corresponds to hierarchical nonlinear tensor decomposition. We propose to solve the learning…
Maximizing margins leads to lossless compression of training data.
problem Generalization in supervised learning.
method Information-theoretic interpretation of margin maximization.
result Margin maximization is a form of lossless maximal compression.
Paper proposes a hybrid model-based and data-driven method for one-bit compressive variational autoencoding.
problem Designing efficient one-bit compressive sensing systems.
method Hybrid model-based and data-driven approach for one-bit compressive variational autoencoding.
result Significant improvement in one-bit compressive sensing compared to state-of-the-art methods.
Paper uses neural networks to compress large portfolios of options, reducing risk and capital requirements.
problem Managing risk and capital requirements for large portfolios of financial options.
method Artificial neural network framework for portfolio compression, static hedging, and risk management.
result The compressed portfolio's risk profiles align closely with the target portfolio's, reducing capital requirements.
WEST compresses word embeddings and softmax layers for memory efficiency.
problem Memory constraints in large vocabulary models.
method WEST encodes words with sequences of sub-units, improving compression without performance loss.
result WEST achieves significant compression without sacrificing performance.
Interpreting black box classifiers, such as deep networks, allows an analyst to validate a classifier before it is deployed in a high-stakes setting. A natural idea is to visualize the deep network's representations, so as to "see what the network sees". In this paper, we demonstrate that standard dimension reduction m…
New method identifies differences between groups in low-dimensional data representations.
problem Identifying meaningful differences between groups in low-dimensional data representations.
method Introduce Global Counterfactual Explanation (GCE) and Transitive Global Translations (TGT) for computing GCEs.
result TGT identifies sparse, accurate explanations that match real data patterns.
VIBI interprets black-box systems by selecting key features that are both brief and comprehensive.
problem Lack of concise and comprehensive explanations for black-box decision systems.
method VIBI uses the information bottleneck principle to select key features that are maximally compressed and informative.
result VIBI provides more concise and comprehensive explanations compared to existing methods.
A tutorial on VAEs explaining its derivation and applications.
problem Understanding the VAE model and its limitations.
method Explains VAE through probabilistic and information theoretic perspectives.
result Identifies two common misconceptions and their practical consequences.
Spectral mixture (SM) kernels comprise a powerful class of generalized kernels for Gaussian processes (GPs) to describe complex patterns. This paper introduces model compression and time- and phase (TP) modulated dependency structures to the original (SM) kernel for improved generalization of GPs. Specifically, by adop…
New approach selects sparse features without validation.
problem Overfitting and high storage/compute costs in AI models.
method Phase transition in probability of feature retrieval.
result Flexible approach for various models and loss functions.
Method reconstructs hidden Markov chains from insurance data.
problem Recovering hidden Markov chains from incomplete insurance data.
method Neural architecture to explicitly provide transition probabilities.
result Neural model successfully validates decompression of insurance information.
A new method clusters covariates considering class labels for better classification.
problem Clustering covariates independently of class labels can lead to poor results.
method Formulates as convex optimization, uses ADMM for solving, and selects model via marginal likelihood.
result Proposed method offers a unique global minimum and improves classification.
We apply L0 regularization to reduce neural network complexity and interpretability.
problem Over-parameterization in neural networks leads to susceptibility to attacks, loss of interpretability, and increased SWaP-C.
method We use L0 regularization to reduce complexity and evaluate the trade-off between complexity and desired metrics.
result L0 regularization captures saliency in the input space and reduces complexity of neural networks.
Pruning neural networks reduces parameters without sacrificing interpretability.
problem Reducing unnecessary structure in neural networks to improve efficiency.
method Examined the effect of pruning on the number of hidden units learning disentangled representations.
result Pruning does not harm interpretability until a significant portion of parameters are removed.
ReduNet optimizes data compression by maximizing rate reduction in deep networks.
problem Optimizing deep networks for high-dimensional multi-class data.
method Maximizing rate reduction through iterative gradient ascent, leading to a multi-layer deep network.
result ReduNet achieves optimal linear discriminative representation and is more efficient in the spectral domain.
New adapted renormalized volume for hyperbolic 3-manifolds with compressible boundary.
problem Analyzing convex co-compact hyperbolic 3-manifolds with compressible boundaries.
method Defining and analyzing a new version of the renormalized volume.
result The adapted renormalized volume is bounded and has properties analogous to the classical renormalized volume.
Long Short-Term Memory (LSTM) is one of the most widely used recurrent structures in sequence modeling. It aims to use gates to control information flow (e.g., whether to skip some information or not) in the recurrent computations, although its practical implementation based on soft gates only partially achieves this g…
Often the analysis of time-dependent chemical and biophysical systems produces high-dimensional time-series data for which it can be difficult to interpret which individual features are most salient. While recent work from our group and others has demonstrated the utility of time-lagged co-variate models to study such …