arXiv preprint ·
Adapting Open-Weight MLLMs to Generate Point Prompts for Electron Microscopy Segmentation
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Promptable models such as microSAM segment electron microscopy (EM) images from point prompts, but automation requires generating prompts without user input. We ask whether open-weight multimodal large language models (MLLMs) can generate them from natural-language requests by returning coordinates to a frozen segmenter. To that end, we convert masks from three mitochondria datasets into training examples, pairing images and instructions with centroid coordinates, then train LoRA adapters while freezing the MLLM backbone and microSAM. We find that Qwen3-VL reaches segmentation AP$_{50}$ $0.736$ after supervised fine-tuning and reward optimization, up from $0.247$ without adaptation, while automatic prompt generation (APG) achieves $0.773$. In addition, two other MLLMs improve, reaching or exceeding APG. When compared with a supervised centroid-heatmap detector that reaches AP$_{50}$ $0.904$ for this mitochondria task, Qwen3-VL more closely matches the annotated point set and instance counts. Moreover, training on two public datasets transfers to an unseen third, while training on all three transfers to an independent EM volume. Robustness tests show stable performance under unseen formulations of the natural-language request, while the coordinates can be reused by a second segmenter. To our knowledge, this is the first feasibility study of open-weight MLLMs as EM point generators, providing an inspectable, language-directed link between localization and mask decoding.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computer Vision
ChatGPT Images 2.5 on Forgery Tasks: Testing Advertised Improvements Against Known Answers
Task-Specified Active Metrological Inspection with Measurement-Steered VLA Manipulation and Deterministic Evidence Gating
YOLO12-MambaScan: An Efficient Object Detector with High-Frequency Enhancement and State-Space Modeling
What Input Resolution Is Required for Bird Species Identification, and What Is Its Latency Cost on an Edge Device? A Study of 14 Input Resolutions and Six Architectures with On-Device Measurements
Compositional Shift Algebra: Extrapolating Mixed Robot Shifts Without Mixed Finetuning
Get the few papers worth your time in Computer Vision every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.