Foundation models · Preprint · arXiv:2606.03773 Co-first author

KletterMix

KletterMix: Climbing Toward High-Quality German Pretraining Data – The Full Report

Building stronger German language models starts with better training data.

Read the paper Dataset

Authors

M. Kraus, R. Härle, S. Sztwiertnia, A. G. Khan, M. Ali, M. Fromm, N. Flores-Herr, K. Kersting

THE IDEA, AT A GLANCE

CONCEPTUAL ILLUSTRATION

Start with curated data

Use a diverse English pretraining corpus.

Translate & filter

Preserve document structure and filter by estimated quality.

Validate by training

Test German data in controlled model experiments.

Translate a curated English corpus, score and filter the German translations, and test their usefulness through controlled model training.

Motivation

German pretraining resources are often smaller and less thoroughly curated or tested than English ones. A useful corpus should offer both diverse content and evidence that it helps models learn.

Method

KletterMix translates the English ClimbMix corpus into German while preserving document boundaries and source metadata. Translation-quality estimates train a lightweight proxy for scoring and filtering the corpus. Controlled pretraining and later-stage training experiments test how the resulting data affects model behavior.

Results

The report finds strong estimated translation quality across domains and gains on German evaluations in the tested training settings. Filtering improves the German aggregate in the small-model experiments; larger-model mixture experiments also examine the balance between German gains and English retention.