arXiv preprint ·
When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Sparse mixtures of experts combine multiple prediction models while evaluating only a small subset for each input. Although this architecture reduces computation, how expert separation and routing sparsity jointly govern expert recovery remains an open problem. In this paper, we address this question for location Gaussian mixtures of experts with a fixed number of experts by establishing how expert separation and spatial arrangement determine recovery rates. For dense models, our results reveal that when experts form well-separated clusters, the separation geometry is governed by the size of the largest cluster; without a prescribed cluster structure, this geometry is instead driven by the total number of experts. For sparse models, the separation geometry on retained routing regions is determined by the number of active experts, while the contribution of omitted regions is controlled by their total probability. Consequently, activating fewer experts reduces the separation complexity in these bounds. Furthermore, Top-1 routing yields a particularly striking result: experts can be recovered at a rate independent of the separation quantity. Our technical approach combines polynomial interpolation and Gaussian reproducing-kernel Hilbert space (RKHS) test functions with quantitative restriction inequalities that extend pointwise comparisons to expert functions.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning (Statistics)
Variational learning of energy-stable macroscopic flow equations from microscopic laws
On Stochastic Continuous-Time Linear System Identification for Adaptive Stabilization
Transport for Conditional Sampling: Foundations and Applications
Efficient Proper Agnostic Learning of Halfspaces
Correctness and Predictive Uncertainty in Bayesian TSK Fuzzy Regression: A Reproducible Evaluation on Tabular and Building-Meter Data
Get the few papers worth your time in Machine Learning (Statistics) every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.