FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

Abstract

Federated training of foundation models is constrained by client memory and communication costs. LoRA-based methods reduce these costs through low-rank adapters, but their fixed rank budget can limit adaptation. Gradient low-rank optimization offers greater flexibility, yet independently chosen client subspaces create a problem we term \emph{subspace fragmentation}: local projections interact with data heterogeneity to bias aggregated directions, while aggregation can increase update rank and communication cost. Thus, accurate local gradient compression need not preserve global descent. We propose \texttt{FedLore}, which shares a low-rank optimization basis within each round and refreshes it across rounds. The shared basis enables exact aggregation in low-rank coordinates and eliminates the identified projection bias. Subspace refresh allows the accumulated model update to exceed the per-round rank budget. We characterize the aggregation bias and establish an $O(T^{-1/2})$ stationarity bound for the projected-SGD variant under a global-gradient coverage condition and standard smoothness and variance assumptions, with bounded gradient heterogeneity. Experiments on vision and language tasks, including federated pre-training, show that \texttt{FedLore} outperforms the evaluated low-rank adapter baselines and matches or exceeds full-parameter training, while reducing communication and optimizer-state memory.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Artificial Intelligence

• arXiv preprint · 1 October 2026

  Geometric Similarity in VLM Low-Level Vision Representations

  Shao-Jun Xia, Huixin Zhang, Zhen Lei and 3 others
• arXiv preprint · 2 October 2026

  Time Series Forecasting Benchmarks Need Scenario-Grounded Stress Testing

  Yuyang Zhao, Lian Xu and Hao Xue
• arXiv preprint · 1 October 2026

  Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning

  Pedro Robles Dutenhefner, Dikshant Shehmar, Wagner Meira and 1 other
• arXiv preprint · 2 October 2026

  VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

  Zekai Wang, Yingqiang Ge, Zekun Wang and 6 others
• arXiv preprint · 1 October 2026

  Don't Waste the Noise: Importance-Guided Perturbation Allocation under Joint Global and Local Constraints

  Melika Shirian and Kianoosh Vadaei

All new papers in Artificial Intelligence