Research
On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

60119179238 · Jun 202019922001200920172026
48 results for weight sharding

This paper optimizes deep learning training by efficiently sharding weight updates across replicas.

problem Redundant weight update computation on all replicas in data-parallel training.
method Automatic sharding of weight updates using static analysis and transformations on the training graph.
result Substantial speedups achieved on large-scale models using Cloud TPUs.

A new framework for clustering high-dimensional data using vertical shards.

problem Clustering high-dimensional data with the curse of dimensionality.
method Vertical Consensus Inference (VCI) that splits data into vertical shards for posterior inference.
result VCI can approximate inference on random partitions for high-dimensional data.

GShard enables scaling of large neural networks with automatic sharding and lightweight APIs.

problem Scaling neural networks to handle vast training data and compute efficiently.
method GShard uses lightweight annotation APIs and XLA compiler extensions for parallel computation.
result GShard successfully trained a 600 billion parameter model on 2048 TPUs in 4 days.

We analyze the Hessian spectra of large models up to 100B parameters.

problem Accurate Hessian spectra of large foundation models are difficult to obtain.
method We use shard-local finite-difference Hessian vector products and stochastic Lanczos quadrature.
result We produce the first large-scale spectral density estimates of foundation models.

In federated distributed learning, the goal is to optimize a global training objective defined over distributed devices, where the data shard at each device is sampled from a possibly different distribution (a.k.a., heterogeneous or non i.i.d. data samples). In this paper, we generalize the local stochastic and full gr…

2019-10-31abs ↗pdf ↗

NEST optimizes deep learning training by placing devices efficiently across networks and memory.

problem Inefficient device placement in distributed deep learning leads to high communication and memory overhead.
method NEST uses network-, compute-, and memory-aware dynamic programming to optimize device placement.
result NEST achieves up to 2.43 times higher throughput and better memory efficiency.

New method ensures consistent inference across different tensor parallel sizes for large language models.

problem Non-deterministic inference in large language models due to inconsistent reduction orders across GPUs.
method Tree-Based Invariant Kernels (TBIK) that align intra- and inter-GPU reduction orders through a unified hierarchical binary tree structure.
result Bit-wise identical results across different tensor parallel sizes for RL training.

Adaptive batch size schedules improve language model training efficiency and generalization.

problem Dilemma of choosing batch sizes in large-scale model training.
method General-purpose adaptive batch size schedules compatible with data and model parallelism.
result Adaptive batch size schedules outperform constant batch sizes and heuristic warmup schedules.

A new weighted MCC measure improves classifier performance evaluation.

problem Lack of measures sensitive to observation weights in multiclass classification.
method Proposes weighted versions of Pearson-Matthews Correlation Coefficient (MCC) for binary and multiclass classification.
result Weighted MCC values are higher for classifiers that perform better on highly weighted observations.

Develops theory of weightings for Lie groupoids and algebroids.

problem Understanding differential geometry of weightings for Lie groupoids and algebroids.
method Extending work on weighted manifolds, defining weighted submanifolds, and developing theories of linear weightings and multiplicative weightings.
result Characterizes infinitesimally multiplicative weightings for Lie algebroids and classifies multiplicative weightings of Lie groupoids.

The paper extends spin geometry to weighted manifolds and defines a new mass for Ricci flow.

problem Generalizing spin geometry to weighted manifolds and defining a new mass.
method Investigates spectral properties of the weighted Dirac operator and defines a new mass.
result Defines a new mass for weighted asymptotically Euclidean manifolds and shows its monotonicity under Ricci flow.

The study explores weightings on submanifolds and their geometric properties.

problem Understanding weightings on submanifolds and their geometric implications.
method Detailed exploration of weighted normal bundles, weighted deformation spaces, and weighted blow-ups.
result A description of weightings in terms of subbundles of higher tangent bundles, leading to new concepts for Lie algebroids and groupoids.

Defines and proves properties of weighted renormalized volume coefficients.

problem None explicitly stated; focuses on mathematical definitions and proofs.
method Defines weighted renormalized volume coefficients and proves their variational nature and polynomial representation.
result Weighted renormalized volume coefficients are variational and can be expressed as polynomials of specific tensors.

Method measures weight similarity in neural networks using normalization and statistical inference.

problem Quantifying weight similarity in non-convex neural networks.
method Chain normalization rule and hypothesis-training-testing statistical inference.
result Weights of identical neural networks converge to similar local solutions.

Study on stable minimal hypersurfaces under Ricci curvature constraints.

problem Stability of weighted minimal hypersurfaces under Ricci curvature bounds.
method Derive geometric consequences and prove a Schoen-Yau type criterion.
result Structure theorem for three-dimensional weighted manifolds of non-negative Ricci curvature.

The study establishes comparison theorems for weighted Finsler manifolds and spacetimes.

problem Analyzing weighted Finsler manifolds and spacetimes with curvature conditions.
method Using weight function and εε-range, the Bonnet-Myers theorem, Laplacian comparison theorem, and Bishop-Gromov volume comparison theorem are formulated.
result New comparison theorems for weighted Finsler manifolds and spacetimes are derived, including those for weighted Riemannian manifolds.

The study analyzes weighted manifolds with curvature bounds, proving eigenvalue estimates and inequalities.

problem Analyzing geometric properties of weighted manifolds under Ricci curvature bounds.
method Develops geometric analysis techniques on weighted Riemannian manifolds with lower 00-weighted Ricci curvature bounds.
result Proves eigenvalue estimates for Steklov and ABP inequalities on weighted manifolds.

The paper generalizes K-stability results to singular and weighted settings.

problem Generalizing K-stability to singular and weighted settings.
method Generalization of results in \cite{Li22a} to singular and weighted settings.
result The \(\mathbb{G}\)-uniform weighted K-stability for models implies \(\mathbb{G}\)-coercivity of the weighted Mabuchi functional.

Study on deformation of weighted scalar curvature, proving geometric results and stability.

problem Deformation of weighted scalar curvature and related geometric properties.
method Linearization of weighted scalar curvature, studying kernel of formal adjoint.
result Definition and study of weighted vacuum static spaces, stability results on flat spaces.

The Penrose theorem and Hawking's topology theorem are extended to weighted spacetimes.

problem Extending Penrose's singularity theorem and Hawking's topology theorem to weighted spacetimes.
method Using weighted null energy condition and synthetic dimension to generalize the theorems.
result Generalized versions of the Penrose and Hawking theorems hold under a weighted null energy condition.

The paper predicts edge weights in weighted directed networks using metric geometry.

problem Predicting edge weights in weighted directed networks.
method Introducing new types of weighted directed networks (AWDNs), constructing metrics, and proposing modified kNN and SVM methods.
result The proposed methods outperform traditional approaches in predicting edge weights.

Optimal weight windows are found by projecting the origin onto a convex polytope.

problem Finding the best weight windows for a weighted moving average smoother.
method Formulated as a quadratic program and projection onto a convex polytope.
result Optimal weight windows are symmetrical and decrease in weight away from the center.

Optimizes weights for better model performance in shifting data.

problem Improper importance weighting leads to poor model performance in data shifts.
method Interprets weights as a bias-variance trade-off and optimizes them simultaneously with model parameters.
result Optimizing weights significantly improves model generalization performance.

Study of weighted nonlinear flags in symplectic geometry.

problem Understanding the geometry of weighted nonlinear flags.
method Generalizing weighted nonlinear Grassmannians to Frechet manifolds and using them to describe coadjoint orbits.
result Description of coadjoint orbits of Hamiltonian diffeomorphisms using weighted isotropic nonlinear flags.

Smooth superspace with special weights has a Fubini-Study form.

problem Describing a new smooth superspace with a special structure.
method Construction of weighted projective superspace and description of its structure.
result Smooth superspace with weights +1,1+1, -1 has an analog of the Fubini-Study form.