Foundation models · Technical report · arXiv:2607.09424

Soofi S

A Sovereign, Open-Source Foundation Model for German and English

An efficient bilingual foundation model, built with a focus on German and transparent training.

Read the paper

Authors

Soofi-Team: B. Droste, D. Fitzek, R. Härle, et al.

THE IDEA, AT A GLANCE

CONCEPTUAL ILLUSTRATION

Bilingual pretraining

Build a deliberate German–English data mixture.

An efficient architecture

Use hybrid layers and activate selected experts per token.

Evaluate & document

Compare capabilities and describe the training process.

A bilingual training mixture feeds a hybrid architecture; evaluation and documentation make the resulting model easier to assess.

Motivation

German-language AI needs capable models that can be independently evaluated and deployed. That calls for attention to German training data, efficient inference, and a documented training process.

Method

Soofi S combines Mamba and Transformer components with a Mixture-of-Experts design, activating only a subset of its parameters per token. Its staged pretraining recipe up-weights German relative to the reference model’s mixture. The report documents the training sources, recipe, and evaluation setup.

Results

The report finds competitive German and English performance against the evaluated base models, with strong code results. Its serving tests show a long-context throughput advantage over the compared dense models. Revised headline aggregates exclude GPQA and the withdrawn held-out group following a disclosed contamination incident.