ZeRO optimizes memory for training large models, scaling to trillions of parameters.
problem Training models with billions to trillions of parameters is challenging due to limited device memory.
method ZeRO eliminates memory redundancies in data- and model-parallel training, scaling model size proportional to the number of devices.
result ZeRO trains models of up to 13B parameters without model parallelism, achieving super-linear speedup and throughput of 15 Petaflops.
Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.
problem Understanding the optimal balance between depth and width in self-attention models.
method Theoretical predictions and empirical ablations on networks of varying depths and widths.
result An optimal width of 30K is recommended for a 1-Trillion parameter network, marking a significant width for self-attention models.
Photonic co-processor speeds up training of large neural networks.
problem Training large neural networks with backpropagation is inefficient and communication is a bottleneck.
method Direct Feedback Alignment (DFA) with a photonic accelerator.
result Photonic accelerator can compute random projections with trillions of parameters.
Ultra-fast search algorithm for trillion-scale corpora with semantic flexibility.
problem Efficiently searching over large natural language corpora with semantic variations.
method String matching based on suffix arrays, vector representation of words, dynamic corpus-aware pruning, fast exact lookup.
result Substantially lower search latency compared to existing methods on FineWeb-Edu corpus.
Model estimates foreign exchange reserve compositions of undisclosed central banks.
problem Limited information on central bank reserve compositions hinders analysis.
method Hidden Markov Model relating portfolio valuation to exchange rates.
result China's reserve composition likely matches global average, while Singapore holds fewer US dollars.
Graph embedding methods produce unsupervised node features from graphs that can then be used for a variety of machine learning tasks. Modern graphs, particularly in industrial applications, contain billions of nodes and trillions of edges, which exceeds the capability of existing embedding systems. We present PyTorch-B…
We present a system and a set of techniques for learning linear predictors with convex losses on terascale datasets, with trillions of features, {The number of features here refers to the number of non-zero entries in the data matrix.} billions of training examples and millions of parameters in an hour using a cluster …
We present a system that enables rapid model experimentation for tera-scale machine learning with trillions of non-zero features, billions of training examples, and millions of parameters. Our contribution to the literature is a new method (SA L-BFGS) for changing batch L-BFGS to perform in near real-time by using stat…
This paper takes stock of megaproject management, an emerging and hugely costly field of study. First, it answers the question of how large megaprojects are by measuring them in the units mega, giga, and tera, concluding we are presently entering a new "tera era" of trillion-dollar projects. Second, total global megapr…
The paper simplifies influence computations for large-scale machine learning models.
problem Improving training efficiency and accuracy in large-scale models.
method Study influence functions, define memorization, simplify computations.
result Influence functions can be practical for large-scale models, indicating memorization.
NNs can learn efficient algorithms for certain problems.
problem Learning efficient algorithms for specific problems.
method Recurrent Convolutional Neural Networks (RCNNs) that learn efficiently.
result RCNNs can learn as well as efficient algorithms described by a constant-sized program.
Improved estimation for imbalanced data using log odds correction and optimal sampling.
problem Parameter estimation with nonuniform negative sampling for imbalanced data.
method Derive asymptotic distribution of IPW estimator, derive optimal sampling probability, propose likelihood-based estimator.
result Improved estimator has the smallest asymptotic variance.
VMoER improves uncertainty quantification in MoE layers for scalable foundation models.
problem Uncertainty quantification in large-scale models like MoE layers.
method Structured Bayesian approach with amortized variational inference over routing logits and temperature parameter inference.
result Improves routing stability, reduces calibration error, and increases AUROC by 12%.
With the large-scale penetration of the internet, for the first time, humanity has become linked by a single, open, communications platform. Harnessing this fact, we report insights arising from a unified internet activity and location dataset of an unparalleled scope and accuracy drawn from over a trillion (1.5$\times…
QA-Token improves tokenization for noisy data, boosting model performance.
problem Tokenization ignores data quality, limiting model effectiveness on noisy corpora.
method QA-Token combines signal quality with vocabulary construction through bilevel optimization and reinforcement learning.
result QA-Token achieves state-of-the-art performance on genomic and financial datasets.
The Economist recently reported that infrastructure spending is the largest it is ever been as a share of world GDP. With $22 trillion in projected investments over the next ten years in emerging economies alone, the magazine calls it the "biggest investment boom in history." The efficiency of infrastructure planning a…
This paper analyzes a novel type of mortality contingent-claim called a ruin-contingent life annuity (RCLA). This product fuses together a path-dependent equity put option with a "personal longevity" call option. The annuitant's (i.e. long position) payoff from a generic RCLA is \$1 of income per year for life, akin to…
New method uses impact IRR to assess impact investments.
problem Determining financial returns of impact investments remains challenging.
method Adapts modern portfolio theory and financial tools to evaluate impact investments.
result Demonstrates the feasibility and utility of impact IRR for optimizing impact investments.
There is intense interest in understanding the stochastic and dynamical properties of the global Foreign Exchange (FX) market, whose daily transactions exceed one trillion US dollars. This is a formidable task since the FX market is characterized by a web of fluctuating exchange rates, with subtle inter-dependencies wh…
Robots' agility in changing terrain helps financial models adapt to market shifts.
problem Challenges in financial market forecasting due to regime switching.
method Adapts pretrained LLMs using intrinsic market rewards and reinforcement learning.
result Significantly improved accuracy in adapting to market regime shifts.
GDP of China is about 11 trillion dollars and GDP of the United States is about 18 trillion dollars. Suppose that we know for the coming years, economy of the US will experience a real growth rate equal to \%3 and economy of China will experience a real growth as of \%6. Now, the question is how long does it take for e…
Large financial dataset tracks FOMC communications and their impact.
problem Understanding how FOMC communications influence financial markets.
method Constructed a large annotated dataset of FOMC speeches, minutes, and transcripts. Developed a hawk-dove classification task. Evaluated various models on the dataset and used RoBERTa-large for monetary policy stance measurement.
result Monetary policy stance measures derived from FOMC documents predict market performance.
When building large-scale machine learning (ML) programs, such as big topic models or deep neural nets, one usually assumes such tasks can only be attempted with industrial-sized clusters with thousands of nodes, which are out of reach for most practitioners or academic researchers. We consider this challenge in the co…
An efficient algorithm optimizes trades across CFMM networks.
problem Optimizing trades through a network of CFMMs for maximum utility.
method Decomposition method to solve the routing problem.
result Significant performance improvements over commercial solvers.
The study explains transformer scaling laws using statistical and approximation theories.
problem Understanding why transformer scaling laws exist for large models trained on low-dimensional data.
method Established statistical estimation and mathematical approximation theories for transformers on low-dimensional manifolds.
result Predicted a power law between generalization error and model and data sizes, with power depending on intrinsic data dimension.
In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intens…
This paper offers a financial economic perspective on the optimal time (and age) at which the owner of a Variable Annuity (VA) policy with a Guaranteed Living Withdrawal Benefit (GLWB) rider should initiate guaranteed lifetime income payments. We abstract from utility, bequest and consumption preference issues by treat…
Private credit markets have expanded significantly, offering unique lending technology to private equity firms.
problem Understanding the growth and characteristics of private credit markets.
method Systematic survey of academic literature, development of integrated theoretical framework, empirical evidence.
result Private credit markets offer a distinct lending technology with higher spreads over syndicated loans.
Large language models struggle with causal relationships, leading to biases and hallucinations.
problem LLMs struggle with true causal relationships, leading to biases and hallucinations.
method Embed causality into LLMs training process at every stage.
result LLMs need to be trained to understand and apply causal knowledge, not just recite it.
JFR-rg model explains Japan's stable debt despite high interest rates and low growth.
problem Understanding Japan's stable government debt despite high interest rates and low growth.
method Formalizes financial repression channels through JFR-rg model, incorporating financial repression bias and exchange-rate channel.
result Identifies Normalization Trap and Captive Financial System Parameter, showing debt dynamics under financial repression.
With globalization, countries are more connected than before by trading flows, which currently amount to at least 36 trillion dollars. Interestingly, approximately 30-60 percent of global exports consist of intermediate products. Therefore, the trade flow network of a particular product with high added values can be re…
Study examines Trump's crypto influence on markets, revealing conflicts and vulnerabilities.
problem Presidential power and cryptocurrency markets during Trump's second term.
method Mixed-methods approach combining quantitative and qualitative data.
result Political-linked digital assets became a distinct class with systemic vulnerabilities.
The Financial Crisis of 2008 is a worldwide financial crisis causing a worldwide economic decline that is the most severe since the 1930s. According to the International Monetary Fund (IMF), the global financial crisis gave impact on USD 3.4 trillion losses from financial institutions around the world between 2007 and …
The introduction of CCPs in most derivative transactions will dramatically change the landscape of derivatives pricing, hedging and risk management, and, according to the TABB group, will lead to an overall liquidity impact about 2 USD trillions. In this article we develop for the first time a comprehensive approach fo…
This study categorizes RWA tokenization challenges and solutions.
problem Navigating the gap between on-chain deterministic code and off-chain probabilistic reality.
method Taxonomy and comparative analysis of RWA protocols, legal and technical standards.
result RWA tokenization requires overcoming legal and technical interoperability issues.
Since beginning of the 2008 financial crisis almost half a trillion euros have been spent to financially assist EU member states in taxpayer-funded bail-outs. These crisis resolutions are often accompanied by austerity programs causing political and social friction on both domestic and international levels. The questio…
Model analyzes mortgage relief during financial hardship.
problem Understanding and optimizing mortgage relief during financial distress.
method Agent-based model of households and servicers.
result Model replicates real-world mortgage studies and provides fine-grained insights.
Paper generates synthetic financial transactions for AML model testing.
problem Insufficient real data for AML model training.
method Agent-based generator calibrated to real transactions.
result Synthetic datasets improve model comparison by providing complete ground truth.
The study constructs models for SOFR term rates using futures data.
problem Disruption of the LIBOR market and lack of liquid SOFR derivatives.
method Dynamic arbitrage-free models using historical SOFR futures prices.
result Shadow-rate extension needed for zero-boundary term rates.
CSLVAE generates large chemical libraries efficiently.
problem Navigating ultra-large combinatorial synthesis libraries.
method Hierarchically-organized database with molecular encoder and decoder.
result Generates valid molecular graphs without autoregression.
New model detects Alzheimer's and severity from speech, cognitive, and language data.
problem Detecting Alzheimer's disease and its severity from multimodal data.
method Multimodal ensemble system using acoustic, cognitive, and linguistic features.
result State-of-the-art accuracy and robustness in AD detection and MMSE score regression.
Paper analyzes bidding strategies in smart grid PDAs, proposing a new method that outperforms existing strategies.
problem Optimizing bidding strategies in smart grid PDAs to balance energy demand effectively.
method Theoretical analysis of double auctions, derivation of best response, and modeling as an MDP.
result Proposed MDPLCPBS strategy outperforms existing strategies in the PowerTAC wholesale market PDA.
PRUDEX-Compass evaluates FinRL methods on 6 axes for financial market investments.
problem Insufficient evaluation of FinRL methods in financial markets.
method Introduces PRUDEX-Compass with 6 axes and 17 measures for evaluation.
result Demonstrates the effectiveness of PRUDEX-Compass on 4 real-world datasets.
This paper uses alternative data to forecast Japanese real estate performance.
problem Accurate rent and price forecasting in Japanese real estate markets.
method Created a comprehensive house price index using over 5 million transactions and economic factors.
result Alternative data variables can forecast real estate performance effectively.
Study analyzes AI's impact on firms, markets, and workers using large language model data.
problem Understanding AI's effect on firms, markets, and workers.
method Used 380 trillion tokens from 400+ large language models to analyze AI's impact.
result Firms with higher AI exposure earn higher returns, creating an AI premium.
Machine learning models outperform traditional actuarial methods in predicting health insurance costs.
problem Improving accuracy in health insurance pricing to identify concession opportunities.
method Developed and evaluated two machine learning models at the patient and employer-group levels.
result Machine learning models outperformed traditional actuarial models by 20% in predicting costs.
The paper proves ADL mechanisms face a trilemma and optimizes them for fairness, revenue, and exchange solvency.
problem The impossibility of a perpetual futures exchange achieving solvency, revenue, and fairness.
method Formal model of ADL, proving trilemma, and analyzing three ADL mechanisms.
result Optimized ADL mechanisms can reduce trader losses while maintaining exchange solvency.
In the last few years, the financial advisory industry has been impacted by the emergence of digitalization and robo-advisors. This phenomenon affects major financial services, including wealth management, employee savings plans, asset managers, etc. Since the robo-advisory model is in its early stages, we estimate tha…