Improved MoE performance through perturbing cosine router.
problem Representation collapse and parameter redundancy in MoE models.
method Least square estimation of cosine router in MoE, followed by noise addition to improve convergence rates.
result Perturbed cosine router leads to polynomial convergence rates for MoE models.
A new router uses attention-based reinforcement learning to solve detailed routing problems efficiently.
problem Solving detailed routing in integrated circuits while adhering to complex design rules.
method Attention-based reinforcement learning applied to track-assignment detailed routing.
result The attention router achieves over 100x acceleration compared to a genetic router without sacrificing solution quality.
Meta-Router optimizes LLM selection using gold-standard and preference-based data.
problem Training a high-quality LLM router with combined data sources is challenging due to bias and scarcity.
method Developed an integrative causal router training framework to correct bias and improve routing accuracy.
result Our approach delivers more accurate routing and improves the trade-off between cost and quality.
CARROT optimizes LLM routing by choosing the cheapest and most accurate model.
problem Optimizing query routing to the most cost-effective LLM for a given task.
method CARROT uses cost and performance estimates to select the best LLM for each query.
result CARROT is minimax optimal, selecting the best LLM for any query.
Modified cosine distance improves similarity performance in data with variance and correlation.
problem Limitations of traditional cosine similarity in random variable spaces with variance and correlation.
method Proposed a variance-adjusted cosine distance metric to overcome limitations of traditional cosine similarity.
result Modified cosine distance shows 100% test accuracy in KNN model on the Wisconsin Breast Cancer Dataset.
DriftMoE adapts to concept drifts in data streams efficiently.
problem Adapting to non-stationary data streams with concept drifts.
method Online Mixture-of-Experts (MoE) architecture with co-training framework.
result DriftMoE achieves competitive results with state-of-the-art stream learning adaptive ensembles.
LMoE uses LLMs to improve stock trading by selecting experts based on textual and price data.
problem Traditional neural network-based router selection in MoE models is suboptimal and ignores textual data.
method Proposes LLMoE, using LLMs as routers to select experts based on historical price data and stock news.
result LLoM outperforms state-of-the-art MoE models and other deep neural network approaches.
This paper explores how MoE layers improve deep learning performance.
problem Understanding the Mixture-of-Experts (MoE) layer in deep learning.
method Formal study of MoE layer's effectiveness and mechanism.
result MoE layer improves performance by leveraging cluster structure and non-linearity.
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change o…
Cosine schedule is optimal for discrete diffusion models.
problem Choosing the best discretization schedule for diffusion models.
method Optimized using Fisher-Rao geometry.
result Cosine schedule is Fisher-Rao optimal.
Derives hyperbolic laws of cosines and sines with fermionic corrections.
problem Deriving hyperbolic laws of cosines and sines with new mathematical corrections.
method Using Minkowski supergeometry, the laws of cosines and sines are derived in the super hyperbolic plane.
result Identical formulae to classical cases with fermionic corrections for cosines and sines.
Cosine similarity can force points to grow in magnitude, causing convergence issues.
problem Cosine similarity loss can lead to convergence issues in deep learning.
method Analyzing under-explored settings and proposing cut-initialization.
result Cosine similarity optimization forces points to grow in magnitude, leading to convergence issues.
The Möbius energy is one of the knot energies, and is named after its Möbius invariant property. It is known to have several different expressions. One is in terms of the cosine of conformal angle, and is called the cosine formula. Another is the decomposition into Möbius invariant parts, called the decomposed Möbius e…
We study the rigidity of polyhedral surfaces using variational principle. The action functionals are derived from the cosine laws. The main focus of this paper is on the cosine law for a non-triangular region bounded by three possibly disjoint geodesics. Several of these cosine laws were first discovered and used by Fe…
New method for European option pricing faster and more robust.
problem Pricing European options efficiently and accurately.
method Fourier cosine series expansions for models with known characteristic functions.
result More robust and faster than the original COS method.
New optimizer designs respect symmetry, improving deep learning models.
problem Optimizers lack respect for symmetry in neural networks.
method Introduce symmetry-compatible principle for optimizer design.
result Symmetry-compatible optimizers improve model performance.
Feedback alignment methods need to be evaluated for accuracy and gradient cosine similarity.
problem Evaluating feedback alignment methods
method Proposed diagnostic evaluation protocol
result Identified silent failures in standard reporting pair
This paper tackles noise in raw datasets to improve representation learning efficiency.
problem Noise in real-world datasets degrades representation learning quality.
method Proposes denoising Cosine-Similarity (dCS) loss to learn robust representations.
result Empirical results show the dCS loss outperforms baseline objective functions.
A new method combines classical and machine learning PDE solvers efficiently.
problem Combining classical and machine learning PDE solvers to reduce computational cost and improve accuracy.
method Proposes an approximate greedy router to select solvers at each iteration, mimicking a greedy approach.
result Consistently reduces final error and AUC of the error trajectory compared to single-solver baselines and hybrid approaches.
CWGD measures gradient diversity weighted by curvature, improving SGD convergence.
problem Gradient noise in high-curvature directions is underestimated by standard methods.
method CWGD weights gradient diversity by the inverse square root of the Hessian.
result CWGD-Cosine reduces optimization error by up to 20% compared to standard cosine annealing.
T-PSDA improves speaker recognition accuracy on toroidal submanifolds.
problem Improving speaker recognition accuracy on hypersphere embeddings.
method Extends PSDA to model within and between-speaker variabilities in toroidal submanifolds of the hypersphere.
result T-PSDA achieves accuracy on par with cosine scoring on VoxCeleb and large accuracy gains on NIST SRE'21.
We do further investigation in a certain cosine function defined for smooth Minkowski spaces. We prove that such function is symmetric if and only if the referred space is Euclidean, and also that it can be given in terms of the Gateaux derivative of the norm. As an application we use it to study the ratio between the …
The spherical Radon transform on the unit sphere can be regarded as a member of the analytic family of suitably normalized generalized cosine transforms. We derive new formulas for these transforms and apply them to study classes of intersections bodies in convex geometry.
We extend the Fourier cosine method to discrete probability distributions, achieving faster convergence rates.
problem Extending Fourier cosine method to discrete probability distributions.
method Spectral filters and convergence rates analysis.
result Spectral filters achieve one order faster convergence rates than previously recognized.
Person recognition aims at recognizing the same identity across time and space with complicated scenes and similar appearance. In this paper, we propose a novel method to address this task by training a network to obtain robust and representative features. The intuition is that we directly compare and optimize the cosi…
In this short article, we extend the cosine formula for the Möbius energy to generalized O'Hara energies. The newly derived formula gives us a condition for which the right circle minimizes the energy under the length-constraint. Furthermore, it shows us how far the energy is from the Möbius invariant property.
Paper proposes CNN-LSTM for WiFi indoor localization.
problem Classical WiFi localization methods using only spatial data.
method Combines CNN and LSTM for CSI feature extraction.
result Achieves 2.5~m average localization error with 80% under 4~m.
The study compares Euclidean and cosine distances in medical drug prescription prediction.
problem Comparing Euclidean and cosine distances in medical drug prescription prediction.
method Established geometric properties and compared distances in real-world medical data.
result Different distances lead to different optimizing nonlinear kernel embedding frameworks.
A large body of research into semantic textual similarity has focused on constructing state-of-the-art embeddings using sophisticated modelling, careful choice of learning signals and many clever tricks. By contrast, little attention has been devoted to similarity measures between these embeddings, with cosine similari…
Sensors which use electromagnetic induction (EMI) to excite a response in conducting bodies have long been investigated for subsurface explosive hazard detection. In particular, EMI sensors have been used to discriminate between different types of objects, and to detect objects with low metal content. One successful, p…
Two things seem to be indisputable in the contemporary deep learning discourse: 1. The categorical cross-entropy loss after softmax activation is the method of choice for classification. 2. Training a CNN classifier from scratch on small datasets does not work well. In contrast to this, we show that the cosine loss fun…
Improved text classification performance through conformal transformations of kernels.
problem Text document categorization in high-dimensional spaces.
method Introduced new Gaussian Cosine kernel and two conformal transformations.
result Conformal transformations significantly improve kernel performance, especially for sub-optimal kernels.
The COS method proposed in Fang and Oosterlee (2008), although highly efficient, may lack robustness for a number of cases. In this paper, we present a Stable pricing of call options based on Fourier cosine series expansion. The Stability of the pricing methods is demonstrated by error analysis, as well as by a series …
Improved barrier option pricing in Heston model using COS-BEM method.
problem Efficient barrier option pricing in the Heston model.
method Combining Fourier-cosine series (COS) method with Boundary Element Method (BEM).
result Significant computational efficiency improvement and BEM attractiveness for practitioners.
Paper introduces a new method for efficient portfolio risk quantification.
problem Efficiently quantify risk in large portfolios with many trades and few dominant risk factors.
method Combines Fourier-cosine series with tensor decomposition techniques for dimension reduction.
result Achieves relative errors below 0.1% with significant runtime improvement.
The note evaluates different methods for option pricing using Shannon Wavelets.
problem Efficient computation of Shannon Wavelet coefficients for option pricing.
method Evaluation of cosine expansion, direct algorithms, and Filon quadrature.
result Filon quadrature is more efficient for computing Shannon Wavelet coefficients.
Wolpert's cosine formula on Teichmüller space gives the Weil-Petersson Poisson bracket {lα,lβ} for geodesic length functions lα,lβ of closed curves α,β as the sum of the cosines of the angle of intersection of the associated geodesics. This was recently generalized to Hitchin representations by Labourie. I…
Beta-SOD detects and corrects noisy object re-identification using cosine similarity and Beta mixtures.
problem Noisy object re-identification in image datasets.
method Reframed Re-ID as a similarity task, using Siamese networks and Beta mixture models.
result Superior performance in noisy conditions compared to state-of-the-art methods.
A well-constructed classification model highly depends on input feature subsets from a dataset, which may contain redundant, irrelevant, or noisy features. This challenge can be worse while dealing with medical datasets. The main aim of feature selection as a pre-processing task is to eliminate these features and selec…
CoE modularizes LLMs for scalable, cost-effective AI systems.
problem Challenges in scalability, cost, and customization of monolithic LLMs.
method Composition of Experts (CoE) leverages multiple expert LLMs with a router for dynamic selection.
result CoE achieves superior performance with reduced computational overhead and cost.
iCOS method estimates risk-neutral densities and option prices without model assumptions.
problem Estimating risk-neutral densities and option prices without model assumptions.
method Leverages Fourier-cosine technique using option-implied cosine series coefficients, without model assumptions.
result Effective in extracting information from option prices under various market conditions.
Our work presents extensive empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance. In particular, larger cosine distances between final an…
The paper examines how well node similarities are preserved by random projections in graph embeddings.
problem The preservation of node similarities under random projections in graph embeddings.
method Investigation of dot product and cosine similarity preservation by random projections over graph matrix rows.
result Random projections produce unreliable embeddings for dot product, especially for high-degree nodes.
The main purpose of incremental learning is to learn new knowledge while not forgetting the knowledge which have been learned before. At present, the main challenge in this area is the catastrophe forgetting, namely the network will lose their performance in the old tasks after training for new tasks. In this paper, we…
MM-DREX adapts LLM experts for financial trading via dynamic routing.
problem Challenges of non-stationary financial markets and static expert designs.
method MM-DREX uses a VLM-powered dynamic router to allocate expert weights and designs heterogeneous trading experts.
result Significantly outperforms 15 baselines across key metrics.
Paper proposes TRA to learn multiple stock trading patterns.
problem Inconsistent i.i.d. assumption limits stock prediction performance.
method TRA architecture with Optimal Transport for pattern assignment.
result Improves information coefficient (IC) by 0.04-0.06 compared to baselines.
Paper explains contrastive learning using cosine similarity and proposes mitigations for batch size effects.
problem Understanding and improving contrastive learning through batch size effects.
method Unified framework of cosine similarity, theoretical insights, and auxiliary loss.
result Performance improvement in small-batch settings through proposed auxiliary loss.