arXiv preprint ·
Dynamic Positional Attention Modulation for Parameter-Efficient Fine-Tuning of Large Language Models
- Published
- Source
- arXiv preprint, posted before peer review
- DOI
- 10.1145/3770855.3817911
- Original
- Read it at arXiv
- Topics
Abstract
Parameter-efficient fine-tuning (PEFT) has become a standard approach for adapting large language models to downstream tasks. However, most existing PEFT methods rely on uniform and static adaptations, without accounting for the structured heterogeneity of attention across dimensions, heads, layers, and input tokens. In practice, attention representations exhibit non-uniform behavior, and positional encoding mechanisms such as rotary positional embeddings (RoPE) induce dimension-dependent positional structure, making uniform adaptation suboptimal. In this work, we propose DyPAM (Dynamic Positional Attention Modulation), a PEFT method that adapts how positional information contributes to attention by operating directly on the query and key representations. DyPAM combines input-conditioned, dimension-wise modulation with head-wise and layer-wise structural modulation, performing fine-grained adaptation of positional attention aligned with the RoPE-induced structure without modifying the pretrained backbone. Extensive experiments on mathematical and commonsense reasoning benchmarks across multiple backbone models demonstrate that DyPAM consistently outperforms existing strong PEFT baselines.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
Disentangling Models from Personas in Heterogeneous LLM Simulations
Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution
Quality-Aware Self-Correcting Speech Translation on an Edge Device
Dialect-Robust Speech Language Models with Synthetic Pseudo-Dialect Augmentation
HouseholdBench: Evaluating Large Language Models as Predictors of Household Economic Behavior
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.