M2G-LLM: Enhancing Clinical Prediction via Multimodal Graph Reasoning and LLM Context Injection

Abstract

Integrating diverse data modalities --- such as clinical notes, laboratory results, and medical imaging --- is essential for advancing clinical decision-making. While Large Language Models (LLMs) have shown remarkable performance in processing unstructured clinical text, their limited capacity to incorporate non-text modalities hinders their broader utility in healthcare applications. Here, we introduce M2G-LLM (Multimodal MedGraph-LLM), a novel framework that enhances LLMs with multimodal integration and alignment via Graph Neural Networks (GNNs). Our approach models temporal relationships between patient visits, propagates information across clinically similar patients, and aligns heterogeneous data sources to construct enriched multimodal context vectors. These vectors are injected into the intermediate layers of the LLM, enabling joint reasoning over textual and non-textual modalities. We evaluate M2G-LLM on the MIMIC-IV and MIMIC-CXR datasets, demonstrating improvements in clinical prediction tasks over strong baseline models. Our results highlight the promise of combining the language understanding of LLMs with the relational reasoning capabilities of GNNs for comprehensive, multimodal healthcare analysis.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning

• arXiv preprint · 19 September 2026

  SPIBER: Reconstructing Free Energy Landscapes from Short, Unconverged Trajectories with Generative Flow Networks

  Venkata Sai Sreyas Adury and Pratyush Tiwary
• arXiv preprint · 18 September 2026

  TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching

  Zhihao Shu, Md Musfiqur Rahman Sanim, Jie Hu and 4 others
• arXiv preprint · 19 September 2026

  Mask-Aware Execution for Efficient JEPA Training

  Md Musfiqur Rahman Sanim, Zhihao Shu, Bahram Afsharmanesh and 3 others
• arXiv preprint · 18 September 2026

  Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks

  Adrien Deliège, Claas Beger, Marc Van Droogenbroeck and 1 other
• arXiv preprint · 19 September 2026

  StateMem: Single-State Residual Memory with Adaptive Inference for Vision-Language-Action Policies

  Wenzhuo Li, Qiongfeng Shi and Yi Zhou

All new papers in Machine Learning