Contrastive Attention Mitigates Spectral Bias in Spiking Transformers

Abstract

Spiking Transformers merge the energy-efficiency of spiking neural networks (SNNs) with the representational power of self-attention, creating a promising architecture for high-performance, energy-efficient computation. However, a performance gap persists versus its counterparts in artificial neural networks (ANNs). Unlike prior works attributing this to binary activations, we reveal that both spiking neurons and spiking self-attention (SSA) act as low-pass filters through multiscale spectral analysis. This characteristic leads to the dissipation of high-frequency components. To address this issue, we propose the Spiking Contrastive Attention (SCA) paradigm, which draw inspiration from the edge-detection and differential sensing properties of biological visual system. By extracting contrast prototypes via global contrastive aggregation and applying local differential refinement, SCA effectively enhances high-frequency information. Extensive experiments show that SCA is a general module that consistently boosts Spiking Transformers across image classification, semantic segmentation, and event-based tracking. Furthermore, it achieves lower complexity, offering superior efficiency over original SSA. These results establish its potential as a fundamental building block for energy-efficient Spiking Transformers.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Artificial Intelligence

• arXiv preprint · 1 October 2026

  Geometric Similarity in VLM Low-Level Vision Representations

  Shao-Jun Xia, Huixin Zhang, Zhen Lei and 3 others
• arXiv preprint · 2 October 2026

  Time Series Forecasting Benchmarks Need Scenario-Grounded Stress Testing

  Yuyang Zhao, Lian Xu and Hao Xue
• arXiv preprint · 1 October 2026

  Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning

  Pedro Robles Dutenhefner, Dikshant Shehmar, Wagner Meira and 1 other
• arXiv preprint · 2 October 2026

  VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

  Zekai Wang, Yingqiang Ge, Zekun Wang and 6 others
• arXiv preprint · 1 October 2026

  Don't Waste the Noise: Importance-Guided Perturbation Allocation under Joint Global and Local Constraints

  Melika Shirian and Kianoosh Vadaei

All new papers in Artificial Intelligence