One-shot Distributed Learning via Weighted Averaging with Server-side Information

Abstract

Distributed learning has drawn considerable attention due to its capability of learning from distributed data without aggregating raw observations. However, frequent exchange of model updates can become the main bottleneck of communication. Therefore, one-shot approaches where each worker fits a local model and sends it to the server to aggregate emerge. It is known that averaging can achieve the so-called pooled-data mean squared error (MSE) order of a hypothetical node that has access to data from the server and all the workers, and performs empirical risk minimization (ERM) on it. However, when the number of workers increases too fast compared to the local sample size, nonzero local estimation bias prevents averaging from achieving this order. In this paper, we study whether independent server data can overcome this limitation. We consider $m$ workers with $n$ observations each and a server with $p$ observations, all i.i.d., giving $N=mn+p$ observations in total. Under certain regularity conditions, a pooled-data ERM achieves MSE of order $\mathcal O(N^{-1})$. We analyze deterministic nonnegative averaging of convex-loss M-estimators, with equal worker weights and the server's own estimator included. Under our assumptions, including nonzero local bias of order $n^{-α}$, $α\geq1$, sample-size weighted averaging achieves the pooled-data MSE order whenever $m=\mathcal O(n^{2α-1})$. Above this worker scale, it does so if and only if $p=Ω(nm\cdot m/n^{2α-1})$. Hence, server-only estimation, whose MSE is of order $\mathcal{O}(p^{-1})$, can achieve pooled-data order while sample-size weighting fails. Analytical examples and synthetic experiments illustrate when averaging matches the pooled-data MSE order, complemented by CIFAR-10 experiments comparing aggregation strategies with linear and nonlinear classifiers.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning (Statistics)

• arXiv preprint · 3 November 2026

  Nonparametric GARCH: An Iterated Neural Estimator

  Ruizhi Deng, Guohao Shen and Ngai Hang Chan
• arXiv preprint · 4 November 2026

  Discovering like a scientist: Literature-informed dimensionless learning for physical closure of data-driven equations

  Hao Xu, Tianhao Wu and Dongxiao Zhang
• arXiv preprint · 3 November 2026

  Descending Everest, Part II: Oracle Complexity of Minimizing a Sum of Convex $(L_0,L_1)$-Smooth Functions

  Mikhail Rudomanov, Ivan Fomin, Alexander V. Gasnikov and 1 other
• arXiv preprint · 4 November 2026

  Towards the Query Complexity of Sampling under Strong Log-Concavity and Log-Sobolev Inequalities

  Huanjian Zhou and Taiji Suzuki
• arXiv preprint · 3 November 2026

  Statistical Machine Learning for Medicinal Plant Leaves Classification

  Jayani P. Gamage and Thiyanga S. Talagala

All new papers in Machine Learning (Statistics)