Paper presents an unsupervised method for object recognition using pretrained CNN and associative memory.
problem Fine-tuning pretrained CNN models for new domains is time-consuming and requires labeled data.
method Uses a pretrained CNN for feature extraction and a Hopfield network associative memory bank for classification.
result Eliminates the need for backpropagation and achieves competitive performance on unseen datasets.
Model projection transfers convolutional network properties to feedforward networks.
problem Transferring properties between feedforward and convolutional networks.
method Unified node-level framework with tensor-valued activations, model projection.
result Projected CNN nodes inherit GFFN-style trainable structure.
In image classification, visual separability between different object categories is highly uneven, and some categories are more difficult to distinguish than others. Such difficult categories demand more dedicated classifiers. However, existing deep convolutional neural networks (CNN) are trained as flat N-way classifi…
Improved disentanglement in VAEs using aggregated feature maps.
problem Improving disentanglement in Variational Autoencoders (VAEs).
method Regionally aggregated feature maps extracted from pre-trained CNNs on ImageNet.
result 2nd place in NeurIPS 2019 disentanglement challenge.
Automatically detecting sound units of humpback whales in complex time-varying background noises is a current challenge for scientists. In this paper, we explore the applicability of Convolution Neural Network (CNN) method for this task. In the evaluation stage, we present 6 bi-class classification experimentations of …
New CNN learns depth features from scratch, outperforming transfer learning.
problem Limited depth data for RGB-D scene recognition.
method Bottom-up approach combining local weakly supervised training and global fine-tuning, modified CNN architecture.
result State-of-the-art accuracy on NYU2 and SUN RGB-D datasets.
Paper proposes CNN-based time series anomaly detection with transfer learning.
problem Time series anomaly detection in automated monitoring systems.
method CNN for segmentation, transfer learning framework, fine-tuning on unseen classes.
result Successfully tested on multiple synthetic and real data sets.
Paper analyzes weak-to-strong generalization in CNNs, identifying data-scarce and data-abundant regimes.
problem Weak-to-strong generalization in CNNs trained on weak models.
method Formal analysis of gradient descent dynamics in data-scarce and data-abundant regimes.
result Identifies two regimes and distinct mechanisms of generalization in each.
Vision Transformers show different internal representations compared to CNNs.
problem Understanding how Vision Transformers solve image classification tasks.
method Comparative analysis of ViT and CNN architectures on image classification benchmarks.
result ViT has more uniform representations across all layers, while CNNs have more varied representations.
Early diagnosis of interstitial lung diseases is crucial for their treatment, but even experienced physicians find it difficult, as their clinical manifestations are similar. In order to assist with the diagnosis, computer-aided diagnosis (CAD) systems have been developed. These commonly rely on a fixed scale classifie…
Evaluates deep learning models in histopathology for robustness and classification strategies.
problem Lack of comprehensive evaluation of histopathology models beyond accuracy.
method Developed a new methodology to evaluate models on five histopathology datasets, including vision transformers and CNNs.
result Identified insights into cancer classification strategies and robustness against stain variations.
Pyrocast predicts pyrocumulonimbus clouds six hours before they form.
problem Predicting pyrocumulonimbus clouds to adapt to climate change.
method Database of 148 pyroCb events, Random Forests, CNNs, and Auto-Encoders.
result Best model predicts pyroCb with 90% accuracy.
CNN-based prostate cancer grading improves accuracy and efficiency.
problem Manual Gleason grading by pathologists is time-consuming and prone to errors.
method Patch-Based Image Reconstruction (PBIR), Distribution Correction (DC), Quadratic Weighted Mean Square Error (QWMSE).
result Achieved superior expert-level performance (0.8885 quadratic-weighted kappa coefficient).
Hybrid model predicts stock prices with high accuracy.
problem Complex volatility of stock market makes traditional models unsatisfactory.
method Attention-based CNN-LSTM and XGBoost integrated model.
result Hybrid model improves prediction accuracy.
Paper classifies lung nodules in CT scans, improving diagnostic accuracy.
problem Early detection of lung nodules for better treatment planning.
method Proposes four 3D neural networks for direct mapping from 3D images to class labels.
result 3D multi-output DenseNet achieves state-of-the-art classification accuracy.
Task-agnostic data augmentation shows little benefit for pretrained transformers.
problem Evaluating the effectiveness of task-agnostic data augmentation on pretrained transformers.
method Conducted a systematic examination of two data augmentation techniques (Easy Data Augmentation and Back-Translation) across 5 tasks, 6 datasets, and 3 pretrained transformer models.
result Data augmentation techniques previously effective for non-pretrained models fail to consistently improve performance for pretrained transformers, even with limited training data.
Transformer models show robustness across domains with domain adversarial training.
problem Domain adaptation from multiple sources with no labeled data.
method Domain adversarial training and mixture of experts.
result Domain adversarial training improves representation but not performance.
New framework selects high-quality pretraining data without training LLMs.
problem Slow progress in understanding pretraining data due to costly experiments.
method Statistical framework based on perplexity-benchmark correlations.
result Approach outperforms existing methods on multiple benchmarks.
Pretraining models improves text classification accuracy, but diminishing returns are observed with large datasets.
problem Improving text classification accuracy with pretrained models.
method Examined the benefits of pretrained models on text classification tasks with varying amounts of training data.
result As the number of training examples grows into the millions, the accuracy gap between pretrained BERT-based models and vanilla LSTM narrows to within 1%.
Paper studies how few pretraining tasks are needed for a linear model to solve new tasks.
problem How many pretraining tasks are needed for a linear model to solve new tasks?
method Pretrained a linear attention model for linear regression with a Gaussian prior.
result Effective pretraining requires a small number of independent tasks, and the model closely matches Bayes optimal.
Method constructs finance LLMs without instruction data using pretraining and model merging.
problem Developing domain-specific LLMs for finance is resource-intensive.
method Continual pretraining on financial data + model merging of instruction-tuned and domain-specific pretrained vectors.
result Successfully constructs instruction-tuned LLMs for finance without additional instruction data.
Fine-grained pretraining improves neural network's ability to learn rare features.
problem Improving generalization in deep learning models.
method Introducing a hierarchical multi-view structure to confine input data distribution.
result Fine-grained pretraining leads to better accuracy on hard downstream test samples.
Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.
problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.
method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.
result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.
SwishNet improves speech, music, and noise classification and segmentation.
problem Speech, Music, and Noise classification/segmentation for audio processing.
method Proposes a fast and lightweight 1D CNN (SwishNet) for MFCC features, trained with knowledge distillation from ImageNet.
result Achieved high accuracy (>97% clip classification, >93% frame-wise segmentation) on MUSAN corpus.
Paper introduces a multi-stage influence function to track model predictions.
problem Improving natural language processing and computer vision performance.
method Develops a multi-stage influence function to track predictions from finetuned models back to pretraining data.
result Identifies pretraining examples contributing most to finetuning task predictions.
Pretraining method enhances dialogue representation learning across various tasks.
problem Scarce labeled data for specific dialogue tasks.
method Multi-task unsupervised pretraining with natural training objectives.
result Significant improvement in downstream tasks without encoder discrimination.
Measures equivariance in vision models using Lie derivative.
problem Understanding the role of equivariance in recent vision models.
method Introducing Lie derivative to measure equivariance with strong mathematical foundations and minimal hyperparameters.
result Many violations of equivariance can be linked to spatial aliasing in network layers, and larger models tend to display more equivariance.
Synthetic continued pretraining enhances model performance with synthetic data.
problem Data inefficiency in pretrained models when adapting to domain-specific documents.
method Synthetic data augmentation using EntiGraph to create a large synthetic corpus.
result Language models can answer questions and follow instructions without access to domain-specific documents.
WeatherFormer learns robust weather features from small datasets.
problem Modeling complex weather dynamics from limited data.
method Pretrained transformer encoder on large satellite dataset, with spatiotemporal encoding.
result State-of-the-art performance in county-level soybean yield prediction and influenza forecasting.
PruneNet efficiently prunes channels in deep networks, improving accuracy and performance.
problem Improving deep neural network performance and efficiency through channel pruning.
method PruneNet uses a computationally light-weight optimization step to identify and prune channels based on layer redundancy.
result Pruned ResNet models achieve higher accuracy and better performance than non-pruned models.
Theoretical analysis of data quality and synergies in LLMs.
problem Understanding why different training methods require different amounts of data.
method Theoretical analysis of transformers trained on a weight prediction task for linear regression.
result SFT excels on smaller datasets challenging for the pretrained model, while RL benefits from large, not overly difficult data.
Denoised smoothing defends pretrained classifiers against adversarial attacks.
problem Adversarial attacks on pretrained classifiers.
method Prepending a denoiser to any off-the-shelf classifier using randomized smoothing.
result Guaranteed ℓ p \ell_p ℓ p -robustness to adversarial examples without modifying the pretrained classifier. Study shows how pretraining robustness transfers to downstream tasks.
problem Understanding how robustness is transferred from pretraining to downstream tasks.
method Theoretical analysis and practical validation of robustness constraints.
result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.
This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.
problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.
method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.
result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.
New research reveals how the pretraining distribution affects in-context learning in large language models.
problem Understanding how the pretraining distribution influences in-context learning in large language models.
method Developed a theoretical framework to characterize the relationship between pretraining distribution properties and in-context learning performance.
result Characterized a fundamental trade-off between robust task selection and generalization in ICL due to the pretraining distribution's statistical properties.
Research improves federated text models for next word prediction.
problem Training models on distributed devices efficiently and effectively.
method Employ transfer learning in federated learning for next word prediction.
result Enhancements to current baselines with pretrained embeddings and whole model pretraining.
Transformers learn to make decisions in new contexts from offline data.
problem Understanding when and how transformers can perform in-context reinforcement learning.
method Theoretical framework analyzing supervised pretraining for ICRL, including algorithm distillation and decision-pretrained transformers.
result Transformers can efficiently approximate optimal reinforcement learning algorithms for various environments.
This paper enhances language models with knowledge awareness.
problem Understanding how much knowledge pretrained language models grasp.
method Inserting explicit knowledge layers into pretraining without changing transformer architecture.
result Significantly more knowledge packed into transformer parameters.
New study finds best language model architecture and pretraining objective for zero-shot tasks.
problem Evaluating which language model architectures and pretraining objectives best enable zero-shot generalization.
method Compared three model architectures and two pretraining objectives across 170 billion tokens, with and without finetuning.
result Causal decoder-only models trained on autoregressive language modeling exhibit strongest zero-shot generalization.
Publicly pretraining models on Web data may undermine differential privacy.
problem The use of large Web-scraped datasets in differential privacy models.
method Critical review of leveraging pretrained models on public datasets for differential privacy.
result Publicizing pretrained models as 'private' could harm trust and generalize poorly.
Transformers can learn new tasks from diverse pretraining data but struggle with out-of-domain tasks.
problem Transformer models' ability to learn new tasks in-context is limited by their pretraining data coverage.
method Investigation of transformer models trained on ( x , f ( x ) ) (x, f(x)) ( x , f ( x )) pairs, comparing in-context learning capabilities across different task families. result Transformers can identify and learn within task families in their pretraining data but fail with out-of-domain tasks.
Enhances anomaly detection in high dimensions with pretrained networks.
problem Difficult to characterize anomaly in high-dimensional data.
method Residual adaptation to adjust pretrained networks for anomaly detection.
result Significantly outperforms existing methods on anomaly detection benchmarks.
The study examines pretrained models for few-shot image classification.
problem Improving few-shot classification performance with pretrained models.
method Systematic investigation of pretrained models on Imagenet for few-shot image classification.
result Supervised pretrained models outperform unsupervised models in few-shot classification.
This paper proves the theoretical advantage of unsupervised pretraining for machine learning tasks.
problem Understanding why unsupervised pretraining helps in machine learning tasks.
method A generic framework using Maximum Likelihood Estimation (MLE) for unsupervised pretraining and Empirical Risk Minimization (ERM) for downstream tasks.
result Proves an excess risk of i l d e O ( C Φ / m + C Ψ / n ) ilde{\mathcal{O}}(\sqrt{\mathcal{C}_Φ/m} + \sqrt{\mathcal{C}_Ψ/n}) i l d e O ( C Φ / m + C Ψ / n ) for downstream tasks under mild conditions. This paper pretrains actor-critic RL algorithms using expert demonstrations.
problem Stability and efficiency of pretraining methods for actor-critic RL algorithms.
method Employ expert demonstrations in actor-critic reinforcement learning framework, ensuring non-global optimal demonstrations are used.
result Our method outperforms RL algorithms without pretraining and is more simulation efficient.
Paper proposes a pretraining method for soft Q-learning from imperfect demonstrations.
problem Challenges in exploiting expert demonstrations while maintaining exploration potentials.
method γ-discounted biased policy evaluation with entropy regularization.
result Our method effectively learns from imperfect demonstrations and outperforms other methods.
Transformers learn to generalize out-of-distribution with diverse pretraining tasks.
problem Conditions for pretrained transformers to generalize out-of-distribution.
method Empirical study of task diversity and pretraining distribution.
result As task diversity increases, transformers transition from specialized to generalized solutions.
Paper shows pretrained models can categorize Thai social media content effectively.
problem Lack of labeled data for Thai social media content classification.
method Pretrained language models on a large noisy Thai social media corpus, fine-tuned for downstream tasks.
result State-of-the-art results achieved on Thai social text categorization tasks.