Improved MoE performance through perturbing cosine router.
problem Representation collapse and parameter redundancy in MoE models.
method Least square estimation of cosine router in MoE, followed by noise addition to improve convergence rates.
result Perturbed cosine router leads to polynomial convergence rates for MoE models.
A new router uses attention-based reinforcement learning to solve detailed routing problems efficiently.
problem Solving detailed routing in integrated circuits while adhering to complex design rules.
method Attention-based reinforcement learning applied to track-assignment detailed routing.
result The attention router achieves over 100x acceleration compared to a genetic router without sacrificing solution quality.
Meta-Router optimizes LLM selection using gold-standard and preference-based data.
problem Training a high-quality LLM router with combined data sources is challenging due to bias and scarcity.
method Developed an integrative causal router training framework to correct bias and improve routing accuracy.
result Our approach delivers more accurate routing and improves the trade-off between cost and quality.
CARROT optimizes LLM routing by choosing the cheapest and most accurate model.
problem Optimizing query routing to the most cost-effective LLM for a given task.
method CARROT uses cost and performance estimates to select the best LLM for each query.
result CARROT is minimax optimal, selecting the best LLM for any query.
DriftMoE adapts to concept drifts in data streams efficiently.
problem Adapting to non-stationary data streams with concept drifts.
method Online Mixture-of-Experts (MoE) architecture with co-training framework.
result DriftMoE achieves competitive results with state-of-the-art stream learning adaptive ensembles.
LMoE uses LLMs to improve stock trading by selecting experts based on textual and price data.
problem Traditional neural network-based router selection in MoE models is suboptimal and ignores textual data.
method Proposes LLMoE, using LLMs as routers to select experts based on historical price data and stock news.
result LLoM outperforms state-of-the-art MoE models and other deep neural network approaches.
This paper explores how MoE layers improve deep learning performance.
problem Understanding the Mixture-of-Experts (MoE) layer in deep learning.
method Formal study of MoE layer's effectiveness and mechanism.
result MoE layer improves performance by leveraging cluster structure and non-linearity.
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
New optimizer designs respect symmetry, improving deep learning models.
problem Optimizers lack respect for symmetry in neural networks.
method Introduce symmetry-compatible principle for optimizer design.
result Symmetry-compatible optimizers improve model performance.
A new method combines classical and machine learning PDE solvers efficiently.
problem Combining classical and machine learning PDE solvers to reduce computational cost and improve accuracy.
method Proposes an approximate greedy router to select solvers at each iteration, mimicking a greedy approach.
result Consistently reduces final error and AUC of the error trajectory compared to single-solver baselines and hybrid approaches.
Paper proposes CNN-LSTM for WiFi indoor localization.
problem Classical WiFi localization methods using only spatial data.
method Combines CNN and LSTM for CSI feature extraction.
result Achieves 2.5~m average localization error with 80% under 4~m.
CoE modularizes LLMs for scalable, cost-effective AI systems.
problem Challenges in scalability, cost, and customization of monolithic LLMs.
method Composition of Experts (CoE) leverages multiple expert LLMs with a router for dynamic selection.
result CoE achieves superior performance with reduced computational overhead and cost.
MM-DREX adapts LLM experts for financial trading via dynamic routing.
problem Challenges of non-stationary financial markets and static expert designs.
method MM-DREX uses a VLM-powered dynamic router to allocate expert weights and designs heterogeneous trading experts.
result Significantly outperforms 15 baselines across key metrics.
Paper proposes TRA to learn multiple stock trading patterns.
problem Inconsistent i.i.d. assumption limits stock prediction performance.
method TRA architecture with Optimal Transport for pattern assignment.
result Improves information coefficient (IC) by 0.04-0.06 compared to baselines.
Deep learning model classifies concurrent human interactions from WiFi data with high accuracy.
problem Classifying concurrent human interactions from WiFi data with high accuracy.
method Attention-BiGRU deep learning model using Multiple Input Multiple Output radio link.
result Maximum benchmark accuracy of 94% for a single subject-pair, 88% for ten subject pairs.
Fluctuation scaling is observed phenomenon from complex networks through finance to ecology. It means that the variance and the mean of a specific quantity are related as $\ev{σ^2|n}\propto \ev{n|A}^{2α}$ with 1/2≥α≥1 when a parameter A (usually the system size) is varied. A can be the strength of the nod…
PEAR dynamically reconfigures agent roles to prevent persistent biases in multi-agent debates.
problem Persistent positional biases and sensitivity to role assignments in fixed topologies.
method Dynamic reconfiguration of agent roles and sparse topologies based on evolving agent states.
result Significantly improves average accuracy over debate baselines across multiple reasoning benchmarks.
The virtualization of compute and network resources enables an unseen flexibility for deploying network services. A wide spectrum of emerging technologies allows an ever-growing range of orchestration possibilities in cloud-based environments. But in this context it remains challenging to rhyme dynamic cloud configurat…
Paper hypothesizes MLP layers in LLMs can be approximated by sparse Mixture of Experts.
problem Understanding dense MLP layers in LLMs.
method Theoretical connection between MoE models and SAE structure in activation space.
result MLP layers in LLMs can be well approximated by sparse Mixture of Experts.
Study shows multifractality emerging in decentralized cryptocurrency trading.
problem Understanding financial dynamics in decentralized cryptocurrency markets.
method Multifractal Detrended Fluctuation Analysis (MFDFA) on tick-by-tick transaction data.
result Multifractality is emerging in decentralized cryptocurrency trading, with larger fluctuations dominating.
Mobile V-MoEs scale down ViTs for resource-constrained vision tasks.
problem Scaling down Vision Transformers for resource-constrained applications.
method Sparse Mixture-of-Experts (MoEs) applied to entire images, with a stable training procedure.
result Mobile V-MoEs achieve better performance-efficiency trade-offs than dense ViTs.
Proposes differentiable and sparse top-k operators for neural networks.
problem Discontinuity of top-k operator makes it unsuitable for end-to-end training with backpropagation.
method Formulates top-k as a linear program over permutahedron, introduces p-norm regularization, and uses isotonic optimization.
result Successfully applied to neural network pruning, fine-tuning, and routing.
MoE outperforms vanilla networks in detecting latent cluster structures in regression tasks.
problem Detecting latent cluster structures in regression tasks with gradient-based learning.
method Theoretical analysis of Mixture of Experts (MoE) using stochastic gradient descent (SGD).
result MoE can successfully divide the problem into subproblems, allowing each expert to recover simpler functions corresponding to individual clusters.
Developing an Agent-Based Model to Mitigate Adverse Selection in Uniswap v3 Liquidity Providers
problem Adverse selection in Uniswap v3 liquidity providers
method Agent-Based Model incorporating blockchain microstructure and volatility dynamics
result Dynamic fee schedules improve hedged Profit and Loss for liquidity providers
Foundation models outperform supervised methods in time series forecasting across various operational regimes.
problem Lack of domain-specific training and ongoing maintenance in supervised learning for time series forecasting.
method Evaluation of foundation models against standard supervised approaches across four operational regimes: periodic, physically constrained, stochastic, and demand forecasting.
result Foundation models are optimal for cold-start or long-tail scenarios and perform well in domains with transferable periodic structures.
Amortizes MIPS by training neural networks to predict optimal keys.
problem Efficiently solving Maximum Inner Product Search for repeated queries.
method Regression-based SupportNet and KeyNet models trained on support functions.
result Significant improvement in IVF match rates for document embeddings.
A new method for multi-expert learning-to-defer avoids optimization issues.
problem Optimization issues in multi-expert learning-to-defer systems.
method A decoupled surrogate with a softmax classifier head and independent sigmoid heads per expert.
result First multi-expert L2D guarantee with a constant not growing with the expert pool.
Generative model predicts daily activity sequences with duration-aware dynamics.
problem Accurately forecasting granular daily activity sequences for energy demand.
method Hierarchical semi-Markov models with duration-aware dynamics.
result Explicitly modeling activity durations improves predictive performance.
Criminals use malware to disrupt cyber-systems. The number of these malware-vulnerable systems is increasing quickly as common systems, such as vehicles, routers, and lightbulbs, become increasingly interconnected cyber-systems. To address the scale of this problem, analysts divide malware into classes and develop, for…
Population migration is valuable information which leads to proper decision in urban-planning strategy, massive investment, and many other fields. For instance, inter-city migration is a posterior evidence to see if the government's constrain of population works, and inter-community immigration might be a prior evidenc…
A new framework AlphaMix combines multiple trading experts to improve stock investment decisions.
problem Inconsistent financial predictions and lack of model uncertainty in investment decisions.
method Reformulate quantitative investment as a multi-task learning problem, and propose AlphaMix framework.
result AlphaMix significantly outperforms state-of-the-art baselines in financial criteria.
MacroHFT uses memory and context-aware reinforcement learning to improve HFT performance.
problem Overfitting and biased decisions in HFT due to rapid market changes.
method Memory Augmented Context-aware Reinforcement Learning (MacroHFT) that trains multiple sub-agents and a hyper-agent.
result MacroHFT achieves state-of-the-art performance on minute-level trading tasks.
EarnHFT tackles HFT challenges with hierarchical RL, significantly outperforming existing methods.
problem Challenges in applying RL to HFT due to long trajectories and market volatility.
method Three-stage hierarchical RL framework: Q-teacher, diverse RL agents, and minute-level router.
result Significantly outperforms 6 state-of-the-art baselines in profitability.