Ethosoft
Open research · Work in progressTurkish NLP · Tokenization

NEDO Turkish Tokenizer

A morphology-aware tokenizer for Turkish that segments text using root-and-suffix structure rather than frequency statistics alone, built as the first layer of the planned Mercan Turkish model family.

Project record

This page reports the maturity and evidence documented in Ethosoft's working archive. It does not imply peer review, regulatory clearance, or clinical validation.

Methods and evidence

Open research · Work in progressTurkish NLP · Tokenization

NEDO Turkish Tokenizer

A morphology-aware tokenizer for Turkish that segments text using root-and-suffix structure rather than frequency statistics alone, built as the first layer of the planned Mercan Turkish model family.

Approach
Combines Turkish dictionary resources, morphological segmentation, and more than 260 Turkish suffix patterns with vocabulary design aimed at preserving morphological integrity rather than treating agglutinative word forms as arbitrary character spans.
First deployment
Used as the tokenizer for NEDOQwen 0.8B, where tokenizer scope and model evaluation behavior were studied together.
Roadmap
A comparative benchmark against other tokenizers is in preparation, covering tokenization efficiency, morphological-boundary preservation, tokens-per-word cost, segmentation behavior, and downstream LLM training impact.
Program context
NEDO is the first layer of the Mercan model family's planned pipeline: tokenizer → pretraining corpus → SFT/reasoning data → training pipeline → Turkish base models.

Evidence boundary: the tokenizer and its morphological ruleset are released; a completed, published benchmark against comparison tokenizers is still in preparation and not yet part of this record.

Expanded project brief

Starting from how the model reads Turkish, not just which model to use.

Turkish is agglutinative and morphologically rich: a single root can carry many different meanings and grammatical roles through chained suffixes (ev → evler → evlerimiz → evlerimizden). A tokenizer that treats this structure as arbitrary or purely statistical character spans can under-use the morphological information Turkish carries, which affects context-window efficiency, training token counts, inference cost, and how well word forms are represented internally.

NEDO Turkish Tokenizer's approach is to process Turkish words using root, suffix, and morphological structure rather than frequency statistics alone. Development combined Turkish dictionary resources, morphological segmentation, Turkish suffix structures, vocabulary design, token-efficiency evaluation, and language-model compatibility. The current ruleset accounts for more than 260 Turkish suffix patterns alongside a Turkish-lexicon-informed approach to segmentation.

NEDO's first significant deployment was as the tokenizer behind NEDOQwen 0.8B, where tokenizer scope, model evaluation behavior, and Turkish language-modeling problems were studied together. That experience treated the tokenizer as a directly investigable part of model behavior rather than only a data-preparation utility.

NEDO is designed as the first layer of a longer pipeline for the planned Mercan Turkish model family: tokenizer → pretraining corpus → SFT/reasoning data → training pipeline → Turkish base models trained from scratch. A comparative benchmark against other tokenizers—covering tokenization efficiency, morphological-boundary preservation, tokens-per-word cost, segmentation behavior across Turkish text types, and downstream LLM training impact—is in preparation.

Status: the tokenizer and its morphological ruleset are released as open research. The comprehensive comparative benchmark referenced in the project's public materials is still in preparation and is not yet part of this record.