arXiv preprint ·
FedLoRD: Personalized Federated Learning based on Low-Rank Structures of Model Parameter Differences
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
We propose FedLoRD, a personalized federated learning (PFL) method based on the low-rank structure of the differences between the global and personalized model parameters. PFL is a branch of federated learning that aims to learn models tailored to the local data of each client under biased local data distributions. In its local training, each client learns a personalized model by regularizing the difference between its model parameters and the global ones sent from the server. The regularization function needs to be designed with priors on the model parameter difference; however, the existing PFL methods rely on empirical choices (e.g., $\ell_1$- and $\ell_2$-norms), which do not take into account the actual characteristics of the difference. The resulting mismatch of priors can lead to suboptimal personalized models. To address this issue, we propose a PFL method whose regularizer is built on an observed characteristic of the model parameter differences: they are approximately low-rank at every layer. In our method, we first formulate a local training problem with nuclear norm regularization on the layer-wise differences. We further rewrite this problem via the factorized representation of the nuclear norm so that it can be efficiently solved by a plain stochastic gradient descent method. Image classification experiments on three benchmark datasets demonstrate that FedLoRD outperforms several existing PFL methods.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Train to Stitch: Topology-Preserving Regularization for Interoperable Representations
ECG-RSI: Recursive Self-Improvement through Targeted Electrocardiogram Waveform Evolution
ONCship: A Large-Scale Underwater Acoustic Corpus for Ship Recognition
Revisiting Error Feedback for Model Quantization
DeepPV-MRF: End-to-End Partial-Volume Fraction Estimation from MR Fingerprinting Time Series
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.