DP-Net uses dynamic programming for efficient deep neural network compression.
problem Efficiently compressing deep neural networks while maintaining accuracy.
method Dynamic Programming for optimal weight quantization and clustering-friendly training.
result Achieves up to 77X compression ratio on Wide ResNet with minimal accuracy loss.
Model compression improves dynamic forecasting ensembles while reducing computational costs.
problem High computational costs and lack of transparency in dynamic forecasting ensembles.
method Model compression applied to dynamic forecasting ensembles of various types of models.
result Compressed models achieve comparable predictive performance and significant computational savings.
Efficiently compress overparameterized deep models by focusing on low-dimensional learning dynamics.
problem Overparameterized models increase computational and memory costs.
method Study of learning dynamics reveals updates occur within a low-dimensional subspace, leading to a compression algorithm.
result Compressed deep linear networks converge faster and yield smaller recovery errors.
Let M be a nontrivial compression body without toroidal boundary components. We study the dynamics of the group of outer automorphisms of the fundamental group of M on the PSL(2,C)-character variety of M.
New study reveals how heavy-tailed SGD dynamics lead to compressible neural networks.
problem Understanding why large neural networks can be compressed effectively.
method Linking SGD dynamics to compressibility properties of neural networks.
result Large step-size/batch-size ratios and overparametrization lead to heavy-tailed SGD dynamics, making networks compressible.
This work proves that large models can be compressed significantly without losing performance.
problem Achieving comparable performance with smaller models and less data.
method Developed a universal compression theory for neural networks and datasets.
result Proved that a generic permutation-invariant function can be compressed into a function of polylogarithmic size with vanishing error.
Paper uses SGLD to recover signals from generative models, proving convergence under mild conditions.
problem Signal recovery from generative priors in compressed sensing.
method Stochastic Gradient Langevin Dynamics (SGLD) for signal recovery.
result SGLD converges to the true signal under mild assumptions on the generative model.
New approach uses compressible dynamics to train deep models efficiently.
problem Efficient training of deep overparameterized models with low-rank structures.
method Leveraging low-dimensional structures and compressible dynamics within model parameters.
result Improved training efficiency and reduced overfitting in language models.
NeuZip compresses neural network weights to save memory during training and inference.
problem Memory constraints in neural network training and inference.
method Entropy-based dynamic weight compression.
result Significant reduction in memory usage without performance loss.
Computational Fluid Dynamics (CFD) is a hugely important subject with applications in almost every engineering field, however, fluid simulations are extremely computationally and memory demanding. Towards this end, we present Lat-Net, a method for compressing both the computation time and memory usage of Lattice Boltzm…
Model compresses event-like contexts using gated surprise signals.
problem Perceiving a dynamic world as organized events.
method Hierarchical, surprise-gated recurrent neural network architecture.
result Achieves best performance on multiple event processing tasks.
LEAD algorithm speeds up decentralized optimization with compression.
problem Slow convergence and stability issues in decentralized optimization with compression.
method Proposes the first linearly convergent decentralized algorithm with compression.
result First consensus error bound for coupled dynamics of primal and dual updates.
Paper extends port-Hamiltonian model to include internal energy for compressible and incompressible flow.
problem Modeling fluid flow dynamics with internal energy and constraints.
method Derived port-Hamiltonian model using interconnection maps and added internal energy and constraint forces.
result Model accurately represents both compressible and incompressible fluid flow.
Sketching reduces data size for accurate spectral estimation.
problem Estimating spectral density from large simulation datasets.
method Sketching for dimensionality reduction and data compression.
result Sketching provides 90% accurate spectral density estimate with 10% data.
Recurrent neural networks (RNNs) achieve cutting-edge performance on a variety of problems. However, due to their high computational and memory demands, deploying RNNs on resource constrained mobile devices is a challenging task. To guarantee minimum accuracy loss with higher compression rate and driven by the mobile r…
Neural networks compress and sample WDN contamination dynamics efficiently.
problem Infrastructure monitoring of complex, networked systems like water distribution networks is expensive and challenging.
method Developed Graph Fourier Transform (GFT) operators and neural networks (NN) for efficient data collection and inference.
result High accuracy reconstruction of contamination dynamics using only 5-10% of the sample set.
Sparsity-based approaches have been popular in many applications in image processing and imaging. Compressed sensing exploits the sparsity of images in a transform domain or dictionary to improve image recovery from undersampled measurements. In the context of inverse problems in dynamic imaging, recent research has de…
Posterior sampling estimator achieves near-optimal recovery guarantees for signals from any prior distribution.
problem Characterizing measurement complexity for signals from any prior distribution, including the entire space.
method Characterization of measurement complexity using posterior sampling estimator for Gaussian measurements and any prior distribution.
result Posterior sampling estimator achieves near-optimal recovery guarantees for signals from any prior distribution, robust to model mismatch.
NAC-FL optimizes model updates in FL systems by adapting compression to network congestion.
problem Federated Learning systems face congestion and delays in data exchanges.
method NAC-FL dynamically adjusts client compression based on network congestion.
result NAC-FL reduces training time and achieves robust performance improvements.
This work analyzes how different forms of compressibility affect adversarial robustness in neural networks.
problem Understanding the interaction between compressibility and adversarial robustness in neural networks.
method Developed a principled framework to analyze the effects of neuron-level sparsity and spectral compressibility on adversarial robustness.
result Identified that different forms of compression can induce highly sensitive directions in the representation space that adversaries can exploit.
LASER compresses recursive model activations by exploiting their low-dimensional structure.
problem Understanding and optimizing the geometric structure of recursive reasoning trajectories.
method Dynamic low-rank basis tracking via matrix-free subspace tracking with a fidelity-triggered reset mechanism.
result Recursive activations occupy a linear, low-dimensional subspace that can be compressed efficiently.
In this paper, we consider linear state-space models with compressible innovations and convergent transition matrices in order to model spatiotemporally sparse transient events. We perform parameter and state estimation using a dynamic compressed sensing framework and develop an efficient solution consisting of two nes…
In this note we survey some recent results for the Euler equations in compressible and incompressible fluid dynamics. The main point of all these theorems is the surprising fact that a suitable variant of Gromov's h-principle holds in several cases.
We study the problem of learning associative memory -- a system which is able to retrieve a remembered pattern based on its distorted or incomplete version. Attractor networks provide a sound model of associative memory: patterns are stored as attractors of the network dynamics and associative retrieval is performed by…
A new algorithm improves Bayesian federated learning by reducing communication overhead.
problem Bayesian federated learning constraints, including privacy, data ownership, and communication overhead.
method Proposes Quantised Langevin Stochastic Dynamics (QLSD) for Bayesian federated learning, using gradient compression and variance reduction techniques.
result Non-asymptotic and asymptotic convergence guarantees for QLSD and its improved versions.
Often the analysis of time-dependent chemical and biophysical systems produces high-dimensional time-series data for which it can be difficult to interpret which individual features are most salient. While recent work from our group and others has demonstrated the utility of time-lagged co-variate models to study such …
Dynamic model pruning improves performance on deep neural networks without retraining.
problem High memory and latency requirements for deep neural networks on low-end devices.
method Dynamic allocation of sparsity pattern and feedback signal to reactivate pruned weights.
result Sparse models achieve state-of-the-art performance with no additional retraining.
This paper analyzes the training dynamics of binary neural networks using information bottleneck.
problem Training binary neural networks is challenging due to discontinuity in activation functions.
method The approach uses the Information Bottleneck principle to analyze BNN training dynamics.
result Training dynamics of BNNs are different from DNNs, with both phases occurring simultaneously.
The Information Plane theory predicts autoencoders do not compress input information.
problem Understanding the training dynamics of hidden layers in autoencoders.
method Derive a theoretical convergence for the Information Plane of autoencoders using a Gram-matrix based mutual information estimator.
result Ideal autoencoders with a large bottleneck layer size do not compress input information, while a small size causes compression only in the encoder layers.
Prototype for adaptive electron microscopy scans reduces dose and time.
problem Reduce electron microscopy scan time and dose with minimal loss.
method Adaptive partial scanning with reinforcement learning.
result Reinforcement learning trained neural network optimizes scan paths.
Product Kanerva Machines dynamically combine smaller models for better memory organization.
problem Limited organization in the Kanerva Machine.
method Introducing Product Kanerva Machines that dynamically combine multiple smaller Kanerva Machines.
result Product Kanerva Machines can discover spatial tunings that approximately factorize simple images by object.
We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our interest is in describing their convergence with sequence length, assuming no limit…
Efficiently price high-dimensional Bermudan options using tensor compression.
problem High-dimensional option pricing with computational complexity.
method Hierarchical tensor compression for Monte Carlo and dual martingale methods.
result Tensor compression alleviates the curse of dimensionality for Bermudan option pricing.
Softmax temperature influences model representation rank and performance.
problem Understanding and optimizing softmax function's impact on model representations.
method Investigated softmax function's role in deep neural networks, introduced rank deficit bias.
result Softmax temperature affects model representation rank and can improve performance.
Deep convolutional neural networks (CNNs) are powerful tools for a wide range of vision tasks, but the enormous amount of memory and compute resources required by CNNs pose a challenge in deploying them on constrained devices. Existing compression techniques, while excelling at reducing model sizes, struggle to be comp…
Analysis of Vlasov plasma dynamics using matched pair Lie-Poisson formulation.
problem Understanding the dynamics of Vlasov plasma and its kinetic moments.
method Hamiltonian (Lie-Poisson) analysis and matched pair decomposition.
result Observation of mutual interactions between subdynamics in Vlasov plasma.
Optimizes neural network training by dynamically updating Tucker decomposition ranks.
problem Redundant parameters in neural network architectures.
method Geometry-aware training of factorized layers in tensor Tucker format.
result Optimal locally approximating the original dynamics without initial rank knowledge.
Recently, researchers proposed various low-precision gradient compression, for efficient communication in large-scale distributed optimization. Based on these work, we try to reduce the communication complexity from a new direction. We pursue an ideal bijective mapping between two spaces of gradient distribution, so th…
To improve the execution speed and efficiency of neural networks in embedded systems, it is crucial to decrease the model size and computational complexity. In addition to conventional compression techniques, e.g., weight pruning and quantization, removing unimportant activations can reduce the amount of data communica…
In many compressive sensing problems today, the relationship between the measurements and the unknowns could be nonlinear. Traditional treatment of such nonlinear relationships have been to approximate the nonlinearity via a linear model and the subsequent un-modeled dynamics as noise. The ability to more accurately ch…
CSGM framework applied to clinical MRI data for robust reconstructions.
problem Applying deep generative priors to clinical MRI data for high-quality reconstructions.
method Training a generative prior on brain scans from the fastMRI dataset and using Langevin dynamics for posterior sampling.
result Posterior sampling via Langevin dynamics achieves high quality reconstructions in clinical MRI data.
We propose and analyze an online algorithm for reconstructing a sequence of signals from a limited number of linear measurements. The signals are assumed sparse, with unknown support, and evolve over time according to a generic nonlinear dynamical model. Our algorithm, based on recent theoretical results for ℓ1-$…
DGNet solves complex dynamical systems with neural networks and constraints.
problem Real-time accurate solutions for large-scale complex systems.
method Model-constrained discontinuous Galerkin Network (DGNet) for compressible Euler equations.
result DGNet achieves out-of-distribution generalization and improved stability.
Tensor decompositions are powerful tools for large data analytics as they jointly model multiple aspects of data into one framework and enable the discovery of the latent structures and higher-order correlations within the data. One of the most widely studied and used decompositions, especially in data mining and machi…
Paper develops a new fluid flow model with energy exchange through boundaries.
problem Modeling ideal fluid flow with energy exchange through boundaries.
method Port-Hamiltonian model based on Stokes-Dirac structures.
result Wide range of fluid dynamical systems can be achieved with this model.
Paper proposes DCT for efficient hybrid parallel training of large recommendation models.
problem Training large recommendation models at scale with efficient communication.
method Dynamic Communication Thresholding (DCT) for both Data Parallelism and Model Parallelism.
result Reduces communication by 100x and 20x during DP and MP, respectively, improving training time by 37%.
This paper describes a new online convex optimization method which incorporates a family of candidate dynamical models and establishes novel tracking regret bounds that scale with the comparator's deviation from the best dynamical model in this family. Previous online optimization methods are designed to have a total a…
In this paper we present a connection between two dynamical systems arising in entirely different contexts: one in signal processing and the other in biology. The first is the famous Iteratively Reweighted Least Squares (IRLS) algorithm used in compressed sensing and sparse recovery while the second is the dynamics of …