arXiv preprint ·
TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling
- Published
- Source
- arXiv preprint, posted before peer review
- DOI
- 10.18653/v1/2025.findings-emnlp.1053
- Original
- Read it at arXiv
- Topics
Abstract
The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP) tools. We present TransBERT, a novel framework for pre-training language models using exclusively synthetically translated text, and introduce TransCorpus, a scalable translation toolkit. Focusing on the life sciences domain in French, our approach demonstrates that state-of-the-art performance on various downstream tasks can be achieved solely by leveraging synthetically translated data. We release the TransCorpus toolkit, the TransCorpus-bio-fr corpus (36.4GB of French life sciences text), TransBERT-bio-fr, its associated pre-trained language model and reproducible code for both pre-training and fine-tuning. Our results highlight the viability of synthetic translation in a high-resource translation direction for building high-quality NLP resources in low-resource language/domain pairs.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
Matryoshka attribution: Learning to attribute language model outputs to representations and weights
Realize What Matters: Principled Context Representation for Large-Scale Reasoning
Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
Beyond Overlap: Estimating the Causal Effect of Benchmark Exposure
Rewired or Gated? How Instruction Tuning Shapes Knowledge-Conflict Circuits in LLMs
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.