High error rates improve neural network performance and reduce power consumption.
problem Training neural networks with high Bit Error Rates (BERs).
method Trained three Binarized Convolutional Neural Network architectures on various datasets with high BERs.
result High BERs do not significantly degrade test accuracy, enabling more efficient hardware.
RAMs improve GAMs' accuracy by fitting components to subregions of feature space.
problem Subpar accuracy in GAMs due to inability to capture feature interactions.
method Identify subregions of feature space where interactions are minimized, fitting one component per subregion.
result RAMs offer improved expressiveness compared to GAMs while maintaining interpretability.
Dropout and RaM become equivalent in large ResNets as depth and width increase.
problem Improving performance in deep learning models.
method Comparing Dropout and Random Gradient Masking in ResNets.
result Dropout and RaM converge to the same large-scale limiting dynamics in ResNets.
A taxonomy classifies memory networks based on their memory organization.
problem Classifying and understanding the expressive power of different memory networks.
method Developed a taxonomy including RNN, LSTM, neural stack, and neural RAM, analyzing their differences and commonality.
result Showed the relative expressive power of memory networks and how they relate to specific tasks.
This paper optimizes SMPC for neural network inference, reducing memory and time.
problem Memory and time constraints in secure neural network inference.
method Implemented ABY2.0 protocol, optimized memory usage, and used a helper node.
result MNIST inference reduced from 8.03 GB RAM and 200s to 0.2 GB RAM and 32s.
Improves bit error tolerance in RRAM-based BNNs without overfitting.
problem Bit errors in RRAM-based BNNs reduce accuracy and overfit to training error rates.
method Proposes straight-through gradient approximation and a novel regularizer.
result Improves BNNs' robustness to bit errors without overfitting.
New method optimizes memory usage in neural networks, improving sequential learning.
problem Current memory models in neural networks waste memory and computation.
method Formulated an optimization problem to maximize information storage, introduced Cached Uniform Writing.
result Proved Cached Uniform Writing optimizes memory usage and outperforms other methods.
Federated Learning tackles limited user participation with a new risk-aware approach.
problem Limited availability of users in federated learning environments.
method Random Access Model (RAM) and Conditional Value-at-Risk (CVaR) to design a risk-aware federated learning algorithm.
result The proposed approach achieves significantly improved performance under various setups compared to standard federated learning.
RAM extends attention-based mechanism for existence determination.
problem Binary determination of object existence in cluttered images.
method Recurrent attention model (RAM) with k k k -maximum aggregation layer and new reward mechanism. result Significant efficiency and accuracy improvement over existing approaches.
This paper investigates compression techniques for deep neural networks to reduce their size without sacrificing performance.
problem Compression of large deep neural networks for resource-limited platforms.
method Weight pruning, quantization, and lossless weight matrix representations based on source coding.
result Achieved up to 165 times compression rate while maintaining or improving model performance.
The paper studies causal effects of multiple treatments in healthcare databases with rare outcomes.
problem Estimating causal effects of multiple treatments in healthcare databases with rare outcomes.
method The paper designs three sets of simulations and compares the operating characteristics of three types of methods: Bayesian Additive Regression Trees (BART), regression adjustment on multivariate spline of generalized propensity scores (RAMS), and inverse probability of treatment weighting (IPTW) with multinomial logistic regression or generalized boosted models.
result BART and RAMS provide lower bias and mean squared error compared to IPTW methods.
VIBNN accelerates Bayesian Neural Networks on FPGAs for efficient inference.
problem Overfitting and small-data training issues in BNNs.
method Hardware accelerator design for variational inference on BNNs, using novel Gaussian random number generators.
result VIBNN achieves high throughput and energy efficiency on FPGA, matching software performance.
No floating point, no multiplications, no problem! Training efficient networks for resource-constrained devices.
problem Designing efficient neural networks for resource-constrained devices without floating-point operations.
method Discretizing both in-network non-linearities and network weights to avoid floating-point and multiplication operations.
result Training networks without floating-point operations can achieve comparable performance to those using floating-point operations, with less memory usage.
Approaches for big data regression using sufficient statistics.
problem Challenges in computing closed-form solutions for large datasets.
method Multiple learning approaches using sufficient statistics arrays.
result Achieves closed-form solutions for multiple models at half the training time of traditional methods.
This paper compares FAISS and FENSHSES for nearest neighbor search in Hamming space.
problem Comparing nearest neighbor search systems in Hamming space.
method Comprehensive evaluations of indexing speed, search latency, and RAM consumption.
result Better understanding of trade-offs between main memory and secondary memory systems.
Develops a Riemannian archetypal analysis for interpretable non-linear data.
problem Limited performance of classical archetypal analysis on non-linear data.
method Riemannian geometry for data-driven pullback, geodesic convex combinations, convex relaxation followed by non-convex refinement.
result Combines interpretability of classical archetypal analysis with expressive power of modern non-linear models.
Exact distributed algorithm trains Random Forest models on very large datasets.
problem Training Random Forest models on extremely large datasets (billions of examples).
method Exact distributed algorithm without approximating best split search.
result Trains Random Forest models on up to 18 billion examples, significantly faster than existing methods.
Improved deep learning model deployment on tiny MCUs with mixed-precision quantization.
problem Memory limitations prevent accurate deployment of DNN models on tiny MCUs.
method Automated mixed-precision quantization using Reinforcement Learning for MCU constraints.
result Mixed-precision models achieve high accuracy with uniform quantization policies.
Bayesian realized EGARCH models improve tail risk forecasting.
problem Forecasting tail risks in financial markets.
method Developed a Bayesian framework for realized EGARCH models, incorporating multiple realized volatility measures and using robust adaptive Metropolis algorithm for estimation.
result Standardized skewed Student-t distribution and sub-sampled realized range models outperform other models in tail risk forecasting.
SNG-DBSCAN clusters data faster with subsampled similarity queries.
problem Efficiently clustering large datasets with DBSCAN's high complexity.
method Subsampled ε ε ε -neighborhood graph for similarity queries. result Subsampling 0.1 % 0.1\% 0.1% of the graph leads to 200x speedup and 250x RAM reduction. The paper explores how contingency-awareness improves exploration in reinforcement learning.
problem Improving exploration in reinforcement learning environments with sparse rewards.
method Developed an attentive dynamics model (ADM) to discover controllable elements of observations and used it for state representation in exploration.
result Combining actor-critic algorithms with count-based exploration using the ADM representation achieved impressive results on Atari games.
Two new data synthesizers generate multidimensional time series for analysis.
problem Evaluate distance functions on high-dimensional time series.
method Proposed two new data synthesizers: CBF and RAM.
result Evaluation of 1-nearest neighbor classifier using DTW on generated datasets.
biglasso solves memory and computation issues for lasso models on large data.
problem Memory and computational limitations in fitting lasso models to large datasets.
method Memory-mapped files, out-of-core computation, efficient feature screening rules.
result biglasso efficiently handles ultrahigh-dimensional, multi-gigabyte data sets.
Big T-Rex solves FDR-controlled sparse regression on laptops with millions of variables.
problem Scalable FDR-controlled variable selection for high-dimensional data.
method Early terminated random experiments with memory-mapping and permutation-based dummy generation.
result Solves FDR-controlled Lasso problems with 5 million variables on a laptop in 30 minutes.
In real world industrial applications of topic modeling, the ability to capture gigantic conceptual space by learning an ultra-high dimensional topical representation, i.e., the so-called "big model", is becoming the next desideratum after enthusiasms on "big data", especially for fine-grained downstream tasks such as …
This paper enables deep network inference on microcontrollers with improved accuracy and reduced memory usage.
problem Deploying deep networks on resource-constrained edge-devices with low memory and computational constraints.
method Mixed low-bitwidth compression, rule-based iterative procedure for bit precision determination, quantization-aware retraining, and integer-only model conversion.
result Improved Top1 accuracy of 68% on a 2MB FLASH memory STM32H7 microcontroller, 8% higher than 8-bit implementations.
SANNS secures k k k -NNS queries and results without revealing them.
problem Securely searching for nearest neighbors without exposing sensitive data.
method Optimized linear scan and clustering-based algorithm with cryptographic primitives.
result SANNS achieves up to 31x faster response times and scales to 10 million entries.
Paper introduces method to make neural networks symmetrical.
problem Creating symmetrical neural networks for data with inherent symmetries.
method Introduces a method for modifying neural networks to enforce equivariance.
result Group convolutional neural networks are a special case of the introduced framework.
Proposes Neural SDE for better model robustness and generalization.
problem Missing regularization mechanisms in Neural ODE networks.
method Integrates various regularization mechanisms via stochastic noise injection.
result Improves robustness and generalization compared to Neural ODE.
Graphs of neural networks are represented to preserve symmetry, improving performance across various tasks.
problem Lack of equivariance in neural network representations of other neural networks.
method Represent neural networks as computational graphs and use graph neural networks to preserve permutation symmetry.
result Single model encodes diverse neural architectures, outperforming state-of-the-art methods.
Convolutional Neural Processes improve data efficiency in neural processes.
problem Improving data efficiency in neural processes for small datasets.
method Convolutional Neural Processes (ConvNPs) improve data efficiency by leveraging translation equivariance and convolutional neural networks.
result ConvNPs enhance the performance of neural processes in small-data problems.
Neural circuits integrate continuous dynamics efficiently.
problem Efficiently integrating continuous neural dynamics for simulation and learning.
method Compact neural circuits for Runge-Kutta and Adams-Bashforth-Moulton methods.
result Equivalence of neural and numerical integration for polynomial systems.
GNPs use graph neural networks to predict target points with uncertainty quantification.
problem Predicting points on graphs with uncertainty.
method Graph Neural Processes (GNP) that operate on graph data, taking context features and outputting a target point distribution.
result GNPs can quantify uncertainty in graph data predictions.
Investigates how neural network graph structure impacts predictive performance.
problem Lack of understanding between neural network graph structure and predictive performance.
method Developed relational graph representation to analyze neural networks, identifying a 'sweet spot' for improved performance.
result Identified a 'sweet spot' in relational graph structure that significantly improves neural network predictive performance.
Neural Tangents simplifies infinite-width neural networks for research.
problem Training and studying infinite-width neural networks.
method High-level API for specifying complex architectures, analytical or gradient-based training, and automatic distribution.
result Analytical training of infinite-width networks and automatic parallelization.
Novel framework explains generalization in deep neural networks.
problem Understanding and improving generalization in deep neural networks.
method Topological Quantum Neural Networks as the semi-classical limit of Deep Neural Networks.
result Demonstrates that the perceptron, viewed as the semi-classical limit, achieves similar results to standard neural networks without training.
This paper shows overparameterized deep neural networks are convex and learn useful features.
problem Analyzing fully trained overparameterized deep neural networks.
method Generalized neural feature repopulation technique.
result Overparameterized deep neural networks are inherently convex and learn useful features.
Augmented Neural ODEs improve stability and expressiveness of ODEs.
problem Limitations of Neural ODEs in representing complex functions.
method Introducing Augmented Neural ODEs that are more expressive and stable.
result Augmented Neural ODEs outperform Neural ODEs in stability, generalization, and computational efficiency.
Investigates neural codes and their embeddings, proving conjectures and introducing new code types.
problem Analyzing neural codes and their embedding dimensions.
method Combinatorial, topological, and algebraic analysis; proving conjectures; introducing new neural code types.
result Proves conjectures about neural codes and their embeddings, introduces new code types.
Neural networks can approximate functions uniformly across various measures.
problem Universal approximation of functions across different probability measures.
method Proving neural networks are dense in Orlicz spaces, extending classical theorems.
result Neural networks uniformly approximate functions for weakly compact families of measures.
Graph Metanetworks process diverse neural architectures efficiently.
problem Processing diverse neural architectures efficiently.
method Builds metanetworks using graph neural networks to process graphs representing input neural networks.
result Proves GMNs are expressive and equivariant to parameter permutation symmetries.
Quadratic models explain neural network behavior during training.
problem Understanding neural network dynamics during training with large learning rates.
method Developed and tested Neural Quadratic Models.
result Neural Quadratic Models exhibit the 'catapult phase' similar to neural networks.
Optimal rates for shallow ReLU networks in nonparametric regression.
problem Approximating smooth and non-smooth functions with shallow ReLU networks.
method Analysis of shallow ReLU k ^k k neural networks, using variation norms and deep learning theory. result Optimal approximation rates for shallow ReLU networks in nonparametric regression.
New metric compares noisy neural trajectories using optimal transport.
problem Existing metrics fail to capture differences in noisy, dynamic neural responses.
method Proposed an optimal transport distance metric for Gaussian processes.
result Metric effectively compares neural dynamics in different systems.
Paper analyzes Q-learning with neural networks, proving a fast convergence rate.
problem Analyzing the convergence rate of neural Q-learning.
method Finite-time analysis of neural Q-learning with a deep ReLU network.
result Neural Q-learning converges to optimal policy with O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) rate. Neural model guides PBE problem solving in programming.
problem Synthesizing programs from example inputs/outputs.
method Uses a neural model to guide miniKanren's constraint logic programming system.
result Synthesizes programs faster and generalizes to larger problems.
The neural tangent kernel equivalence theorem fails in practice.
problem Does the neural tangent kernel (NTK) equivalence theorem hold in practical neural network training?
method Rigorously derived NTK and conducted numerical experiments to evaluate the equivalence theorem.
result Adding a layer to a neural network and the corresponding updated NTK do not yield matching changes in predictor error.
The FAIRnets Ontology makes neural networks findable, accessible, interoperable, and reusable.
problem The resource-intensive training of neural networks and the lack of training data availability.
method Development of FAIRnets Ontology to model neural networks on a meta-level and creation of a knowledge graph (FAIRnets) of over 18,400 neural networks.
result The FAIRnets Ontology and knowledge graph enable the reuse and recommendation of neural networks to data scientists.