A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Model uncertainty obtained by variational Bayesian inference with Monte Carlo dropout is prone to miscalibration. The uncertainty does not represent the model error well. In this paper, temperature scaling is extended to dropout variational inference to calibrate model uncertainty. Expected uncertainty calibration erro…
HET-XL improves heteroscedastic classifiers for large-scale image classification.
problem Scaling heteroscedastic classifiers to handle large numbers of classes and tuning the temperature hyperparameter.
method HET-XL, a heteroscedastic classifier with independent parameter count from the number of classes, learns the temperature hyperparameter directly from training data.
result HET-XL requires 14X fewer additional parameters and performs better than baseline heteroscedastic classifiers on large image classification datasets.
Hyperfitting improves LLM generation quality by enhancing diversity, contrary to simple temperature scaling.
problem Improving open-ended generation quality of LLMs with minimal fine-tuning effort.
method Demonstrates that hyperfitting, a phenomenon where LLMs are fine-tuned to near-zero training loss, enhances generation quality and mitigates repetition.
result Hyperfitting is distinct from temperature scaling and involves a dynamic, context-dependent rank reordering mechanism in the final transformer block.
Recently, Deep Neural Networks (DNNs) have been achieving impressive results on wide range of tasks. However, they suffer from being well-calibrated. In decision-making applications, such as autonomous driving or medical diagnosing, the confidence of deep networks plays an important role to bring the trust and reliabil…
A theory explaining how deep learning works is yet to be developed. Previous work suggests that deep learning performs a coarse graining, similar in spirit to the renormalization group (RG). This idea has been explored in the setting of a local (nearest neighbor interactions) Ising spin lattice. We extend the discussio…
We present the data on wealth and income distributions in the United Kingdom, as well as on the income distributions in the individual states of the USA. In all of these data, we find that the great majority of population is described by an exponential distribution, whereas the high-end tail follows a power law. The di…
Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the exist…
Theoretical understanding of how deep neural network (DNN) extracts features from input images is still unclear, but it is widely believed that the extraction is performed hierarchically through a process of coarse-graining. It reminds us of the basic concept of renormalization group (RG) in statistical physics. In ord…
We describe an adaptation of the simulated annealing algorithm to nonparametric clustering and related probabilistic models. This new algorithm learns nonparametric latent structure over a growing and constantly churning subsample of training data, where the portion of data subsampled can be interpreted as the inverse …
We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization. Similarly, the logarithm in the log loss we use for training is replaced by a low temperature logarithm. By tuning the two temperatures we create loss functions that are non…
This paper measures temperature in agent systems using volatility.
problem How to measure temperature in agent systems.
method Examined an agent system with two decision options in a news environment, established the measurement equation, and outlined the concept of temperature measurement.
result Illustrated a strategy for influencing average opinion in competing subsystems.
Attention temperature improves robustness of ICL in high-dimensional settings.
problem ICL robustness failure under distribution shift in high dimensions.
method Analyzed a Transformer with approximate softmax attention, derived a closed-form error expression, and showed optimal temperature minimizes error.
result Optimal attention temperature minimizes ICL generalization error under distribution shift.
Trends in terrestrial temperature variability are perhaps more relevant for species viability than trends in mean temperature. In this paper, we develop methodology for estimating such trends using multi-resolution climate data from polar orbiting weather satellites. We derive two novel algorithms for computation that …
Optimizes contrastive learning with individualized temperatures for better performance on imbalanced datasets.
problem The common practice of using a global temperature parameter ignores the varying semantic similarity across different anchor data.
method Proposes a new robust contrastive loss inspired by distributionally robust optimization (DRO) and an efficient stochastic algorithm for automatic temperature individualization.
result Our method automatically learns a suitable temperature for each sample, improving performance on imbalanced datasets.
Over the last 23 years, the U.S. Securities and Exchange Commission has required over 34,000 companies to file over 165,000 annual reports. These reports, the so-called "Form 10-Ks," contain a characterization of a company's financial performance and its risks, including the regulatory environment in which a company op…
The forecasting and reconstruction of ocean and atmosphere dynamics from satellite observation time series are key challenges. While model-driven representations remain the classic approaches, data-driven representations become more and more appealing to benefit from available large-scale observation and simulation dat…