CellMSA: Context Modeling for Single-Cell Representation Learning

Abstract

Single-cell transcriptomics enables profiling of cellular states at unprecedented resolution, but its high dimensionality, sparsity, and technical batch effects pose significant challenges for representation learning. Existing single-cell foundation models typically encode each cell independently or only model cells from the same batch for denoising, thereby underutilizing the rich relational information across batches and cell types to model gene expression patterns. We argue that single-cell models can benefit from more informative cell-context modeling. By comparing consistency and variation across cells, models can capture fine-grained gene-gene dependencies associated with cell states, which are essential for learning high-quality representations. Inspired by the use of multiple sequence alignment (MSA) context in protein modeling, we propose CellMSA, a single-cell representation learning framework that introduces an MSA-inspired inductive bias into transcriptomic modeling. For each target cell, CellMSA retrieves relevant cells from different batches and biologically related cell types as context, and summarizes cross-cell patterns into a context-dependent gene-pair representation. This representation is then injected into a pair-aware target-cell encoder for fine-grained representation learning. We pretrain CellMSA on a large-scale human single-cell corpus of approximately 109 million cell observations, including 65.6 million primary observations. Experiments show that our framework consistently outperforms existing methods across multiple benchmarks. Code is available at the following repository: https://github.com/PharMolix/CellMSA.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Genomics

• arXiv preprint · 4 October 2026

  TempoBridge: Source-Conditioned Flow Matching with Optimal Transport Couplings for Single-Cell Population Transitions

  Bowen Han, Lingbei Meng, Shihuan Luo and 7 others
• arXiv preprint · 30 September 2026

  From DNA Design to DNA Slimming: Auditable Agentic Discovery of a Deletion-Only Designer

  Joel Shor
• arXiv preprint · 7 October 2026

  Taxonomic Classification with Tag Arrays

  Travis Gagie and Gonzalo Navarro
• arXiv preprint · 28 September 2026

  GenoMorph: Pathway-Grounded Genomic Disease Reasoning via Adaptive Latent Computation

  Tanmoy Kanti Halder, Akash Ghosh, Arijit Roy and 1 other
• arXiv preprint · 8 October 2026

  Predictive Multiplicity in Cell-Fate Assignment: Label-Free Rashomon Sets and the Limits of Per-Cell Certification

  Arjun Bhupatiraju and Abhiram Bhupatiraju

All new papers in Genomics