---
title: Reimplementing a Pronunciation-Aware Syllable Tokenizer for Nepali
date: 2026-07-26
---

# Reimplementing a Pronunciation-Aware Syllable Tokenizer for Nepali

# Reimplementing a Pronunciation-Aware Syllable Tokenizer for Nepali

Paper: [Pronunciation-Aware Syllable Tokenizer for Nepali Automatic Speech Recognition System](https://aclanthology.org/2023.icon-1.4/) (Ghimire et al., ICON 2023)

The paper proposes a syllable-level tokenizer for Nepali that improves ASR accuracy (8.09% CER vs 9.3% for a character tokenizer) by splitting text into units that actually correspond to how Nepali is *pronounced*, instead of splitting on raw Unicode characters or BPE merges.

I wanted to reproduce it, but the paper's tokenizer depends on a pre-built **syllable dataset** (a lexicon of valid Nepali syllables used for lookup), and that dataset isn't published anywhere. So instead of reproducing the lookup table, I wrote a small rule-based tokenizer that encodes the same linguistic rules directly — no lexicon required. It reproduces the paper's own worked examples exactly.

Two files:
- [`syllable_dataset.py`](./syllable_dataset.py) — Devanagari character classes + a syllable generator
- [`nepali_syllabic_tokenizer.py`](./nepali_syllabic_tokenizer.py) — the tokenizer itself

## 1. Why character/BPE tokenizers break on Devanagari

Devanagari is an *abugida*: a consonant carries an inherent vowel sound (क = "ka"), and small marks called **vowel markers** (matras) attached above/below/beside it change that vowel (कि = "ki", के = "ke"). A **halant** (्) cancels the inherent vowel, letting consonants stack into clusters (क् + ष = क्ष). None of these marks are pronounceable on their own — they only mean something attached to a consonant.

The paper's motivating example is the word **क्रान्ति** ("revolution"). A character tokenizer splits it into individual Unicode codepoints, several of which — the vowel marker ा and the halant ् — carry no sound by themselves. A BPE tokenizer does a bit better by co-occurrence, but still isn't guaranteed to respect syllable boundaries, since it optimizes for corpus statistics, not pronunciation.

What you actually want is: **क्रा**, **न्**, **ति** — three phonetic chunks that map cleanly onto three spoken syllables, each alignable to a slice of audio for ASR training.

## 2. The paper's approach: sliding window + lexicon lookup

The paper first builds a **syllable dataset**: every grammatically valid syllable in Nepali, generated by combining consonants, complex consonants, vowel markers, halant, and other markers according to Devanagari grammar rules.

Tokenization then works like a greedy longest-match parser over that lexicon:

1. Character-tokenize the sentence into a flat list `T = [C1, C2, C3, ...]`.
2. Slide a window of size 4 (long enough to cover any complex syllable) over `T`.
3. Inside the window, build candidate strings from longest to shortest: `{C1C2C3C4, C1C2C3, C1C2, C1}`.
4. Check each candidate against the syllable lexicon, starting with the longest. The first hit is emitted as a token.
5. Advance the window past however many characters were consumed, and repeat.

This is essentially maximal-munch tokenization with a dictionary — the same family of idea as WordPiece/BPE, except the "dictionary" is a hand-generated set of phonetically valid syllables instead of a corpus-derived vocabulary.

The catch: the dictionary is exactly what makes this work, and it isn't public. Without it, step 4 above can't run.

## 3. Reproducing it without a lexicon

Devanagari's structure means you don't actually *need* to look a syllable up to know where it ends — the character classes themselves tell you. `syllable_dataset.py` defines the classes:

```python
VOWELS = ['अ', 'आ', 'इ', 'ई', 'उ', 'ऊ', 'ए', 'ऐ', 'ओ', 'औ', 'ऋ', 'ॠ', 'ऌ']

CONSONANTS = ['क', 'ख', 'ग', ... , 'स', 'ह']  # 32 consonants

HALANT = '्'                                   # cancels the inherent vowel

VOWEL_MARKERS = ['ा', 'ि', 'ी', 'ु', 'ू', 'े', 'ै', 'ो', 'ौ', 'ृ']  # matras

OTHER_MARKERS = ['ं', 'ः', 'ँ']                # anusvara, visarga, chandrabindu

COMPLEX_CONSONANTS = ['क्ष', 'ज्ञ', 'द्ध', 'द्य', 'त्त', 'द्व'] + \
    [c + '्' + 'र' for c in CONSONANTS]        # e.g. क्र, ग्र, त्र — all length 3
```

That last group matters: a handful of consonant clusters (क्ष, ज्ञ, and any *consonant + halant + र*, like क्र or त्र) act as a single phonetic unit even though they're three Unicode characters. These 39 clusters are the only thing that needs to be listed explicitly — everything else follows from the four character classes above.

### The rule

> Scan left to right. A **vowel**, **consonant**, or **complex consonant** always starts a new token. A **vowel marker**, **halant**, or **other marker** never starts a token — it always attaches to whatever token came before it.

That's the entire algorithm, in `nepali_syllabic_tokenizer.py`:

```python
while i < len(text):
    first_three_chars = text[i:i+3]
    char = text[i]

    if char.isspace():
        tokens.append(" ")
        i += 1
        continue

    # complex consonant (क्ष, त्र, ग्र, ...) — 3 chars, one syllable
    if first_three_chars in SyllableDataset.COMPLEX_CONSONANTS:
        tokens.append(first_three_chars)
        i += 3
        continue

    # vowel or plain consonant — starts a new syllable
    if char in SyllableDataset.VOWELS or char in SyllableDataset.CONSONANTS:
        tokens.append(char)
        i += 1
        continue

    # matra / halant / anusvara-visarga-chandrabindu — glue to the last token
    if char in SyllableDataset.VOWEL_MARKERS + [SyllableDataset.HALANT] + SyllableDataset.OTHER_MARKERS:
        tokens[-1] += char
        i += 1
        continue
```

No window, no candidate generation, no dictionary lookup — just "does this character start a syllable, or does it belong to the previous one?"

### Walking through क्रान्ति

| i | char | class | action | tokens so far |
|---|------|-------|--------|----------------|
| 0 | क्र (3-char lookahead) | complex consonant | new token | `['क्र']` |
| 3 | ा | vowel marker | attach to last | `['क्रा']` |
| 4 | न | consonant | new token | `['क्रा', 'न']` |
| 5 | ् | halant | attach to last | `['क्रा', 'न्']` |
| 6 | त | consonant | new token | `['क्रा', 'न्', 'त']` |
| 7 | ि | vowel marker | attach to last | `['क्रा', 'न्', 'ति']` |

Result: `['क्रा', 'न्', 'ति']` — three phonetic syllables, exactly as intended.

## 4. Does it match the paper's own examples?

The paper publishes a worked example (Figure 1) and a results table (Table 3). Running this implementation on the same inputs:

```python
from nepali_syllabic_tokenizer import NepaliSyllabicTokenizer

t = NepaliSyllabicTokenizer()

t.tokenize("क्षेत्रमा कसैका")
# paper (Fig. 1): क्षे  त्र  मा  क  सै  का
# this impl:  ['क्षे', 'त्र', 'मा', ' ', 'क', 'सै', 'का']

t.tokenize("व्यक्तित्वमा प्रभाव पर्ने")
# paper (Table 3): व्, य, क्, ति, त्, व, मा, ' ', प्र, भा, व, ' ', प, र्, ने
# this impl: ['व्', 'य', 'क्', 'ति', 'त्', 'व', 'मा', ' ', 'प्र', 'भा', 'व', ' ', 'प', 'र्', 'ने']

t.tokenize("घाउ लागेको क्षेत्रमा")
# paper (Table 3): घा, उ, ' ', ला, गे, को, ' ', क्षे, त्र, मा
# this impl: ['घा', 'उ', ' ', 'ला', 'गे', 'को', ' ', 'क्षे', 'त्र', 'मा']
```

All three match token-for-token. Tokenization is also lossless — `"".join(tokens)` reconstructs the original text exactly, which matters for aligning tokens back to audio spans in an ASR pipeline.

## 5. What `syllable_dataset.py` is actually for

I initially built `SyllableDataset` to hold a full generated lexicon, the same way the paper's dataset does — combining every consonant/complex-consonant with every vowel marker and marker combination gives **3,303 valid syllables**. `SyllableDataset.get_all_syllables()` will hand you that set if you want it (e.g. for validating a corpus, or building a vocabulary for a downstream model).

But it turned out the tokenizer itself never needs to search that 3,303-entry set. It only ever consults `COMPLEX_CONSONANTS` — 39 entries — to catch the fixed-width consonant clusters, since a 1-character lookahead can't tell "त" (a plain consonant, syllable end) from "त्र" (a complex consonant, keep going) without checking ahead. Every other decision is a plain class-membership check. The lexicon-generation code stayed in the file because it's a useful standalone artifact, not because the tokenizer depends on it.

## 6. Other behavior worth knowing about

**Optional normalization** (off by default) folds phonetically-merged pairs together, e.g. ई→इ, ऊ→उ, श/ष→स, matching pairs that are pronounced identically in spoken Nepali but spelled differently:

```python
t = NepaliSyllabicTokenizer(normalize=True)
t.tokenize("श्री")   # -> ['स्रि']   (श folded to स)
```

**Punctuation and non-Devanagari input.** Punctuation (both ASCII and Nepali, e.g. । , ? !) is replaced with spaces before tokenization. Digits and any other non-Devanagari characters are passed through as individual tokens by default, or dropped if you construct the tokenizer with `remove_non_devanagari=False`.

## 7. What this is not

This reproduces the *tokenizer* from the paper, not the ASR system built on top of it — I haven't trained the CNN+GRU/CTC model from section 4.3 or measured CER/WER, since that requires the Open SLR Nepali speech corpus and a training run. What's here is the piece that the paper argues matters most: turning raw Devanagari text into pronunciation-aligned units, without needing whatever lexicon the original authors used internally.

It's also not a strict superset of the paper's algorithm — the sliding-window lookup approach could in principle absorb exceptions that don't fit these four character classes if the lexicon encoded them. This rule-based version assumes the grammar rules are regular enough to not need exceptions, which held for every example I tried, but I haven't validated it against a large corpus.

## Appendix: Full Source

<details>
<summary><code>syllable_dataset.py</code></summary>

```python
"""
Syllable Dataset Generator for Nepali Devanagari Script
Generates all valid syllables based on Devanagari grammar rules
"""

from typing import Set, List
import itertools


class SyllableDataset:
    """Generate and manage Nepali syllable dataset based on Devanagari grammar"""
    
    # Devanagari character sets
    VOWELS = ['अ', 'आ', 'इ', 'ई', 'उ', 'ऊ', 'ए', 'ऐ', 'ओ', 'औ', 'ऋ', 'ॠ', 'ऌ']
    
    CONSONANTS = [
        'क', 'ख', 'ग', 'घ', 'ङ',
        'च', 'छ', 'ज', 'झ', 'ञ',
        'ट', 'ठ', 'ड', 'ढ', 'ण',
        'त', 'थ', 'द', 'ध', 'न',
        'प', 'फ', 'ब', 'भ', 'म',
        'य', 'र', 'ल', 'व',
        'श', 'ष', 'स', 'ह' # क्ष = 'क' + '्' + 'ष', त्र = 'त' + '्' + 'र', ज्ञ ='ज' + '्' + 'ञ'
    ]

    HALANT = '्'  # Virama/halant marker
    
    # vowel markers,HALANT, OTHER_MARKERS: should be attached to the last consonant (they should not be individual tokens)
    VOWEL_MARKERS = ['ा', 'ि', 'ी', 'ु', 'ू', 'े', 'ै', 'ो', 'ौ', 'ृ']
    
    OTHER_MARKERS = ['ं', 'ः', 'ँ']
    
    NUMBERS = ['०', '१', '२', '३', '४', '५', '६', '७', '८', '९']
    
    INVALID_TOKENS = ['ऌ', 'ऌं', 'ऌः', 'ऌँ', 'ॠ', 'ॠं', 'ॠः', 'ॠँ']

    # Common complex consonants (NOTE: all complex consonants have len. 3)
    COMPLEX_CONSONANTS = [
        'क्ष', 'ज्ञ','द्ध', 'द्य', 'त्त', 'द्व'
    ] + [consonant + '्' + 'र' for consonant in CONSONANTS]
    # excluding 'त्र' (tra),  'श्र' (e.g. in shree) because they would be covered under: consonant + HALANT + 'र'

    
    def __init__(self):
        self.syllables: Set[str] = set()
        self._generate_syllables()
    
    def _generate_syllables(self):
        """Generate all valid Nepali syllables"""
        
        # 1. Single vowels
        self.syllables.update(self.VOWELS)
        
        # 2. Single consonants (with inherent 'a' sound)
        self.syllables.update(self.CONSONANTS)

        # 3. Add pre-defined complex consonants
        self.syllables.update(self.COMPLEX_CONSONANTS)
        
        # 4. Numbers
        self.syllables.update(self.NUMBERS)
        
        # 5. Space and punctuation
        self.syllables.update([' '])  # '।', ',', '.', '?', '!', '-', '\'', '"'
            
        # 6. Consonant/complex consonant + vowel marker
        for c in self.CONSONANTS + self.COMPLEX_CONSONANTS:
            for vm in self.VOWEL_MARKERS:
                self.syllables.add(c + vm)
                
                # e.g. दिँदै
                for om in self.OTHER_MARKERS:
                   self.syllables.add(c + vm + om)
        
        
        
        # 7. Consonant/complex consonant + halant (dead consonant)
        for c in self.CONSONANTS + self.COMPLEX_CONSONANTS:
            self.syllables.add(c + self.HALANT)

        # 8. Consonants/vowels/complex consonants with other markers (anusvara, visarga, chandrabindu)
        for c in self.CONSONANTS + self.VOWELS + self.COMPLEX_CONSONANTS:
            for om in self.OTHER_MARKERS:
                self.syllables.add(c + om)
        
        # Consonant + Halanta + ra (kra, khra, ...)
        for c in self.CONSONANTS:
            self.syllables.add(c + self.HALANT + 'र')
        
        # # Consonant + Halanta + ra + vowel marker (kraa, kri, ...)
        # for c in self.CONSONANTS:
        #     for vm in self.VOWEL_MARKERS:
        #         self.syllables.add(c + self.HALANT + 'र' + vm)
        
    def contains(self, syllable: str) -> bool:
        """Check if a syllable exists in the dataset"""
        return syllable in self.syllables
    
    def get_all_syllables(self) -> Set[str]:
        """Return all syllables in the dataset"""
        return self.syllables.copy()
    
    def save_to_file(self, filepath: str):
        """Save syllable dataset to a file"""
        with open(filepath, 'w', encoding='utf-8') as f:
            for syllable in sorted(self.syllables):
                f.write(syllable + '\n')
    
    @classmethod
    def load_from_file(cls, filepath: str) -> 'SyllableDataset':
        """Load syllable dataset from a file"""
        dataset = cls.__new__(cls)
        dataset.syllables = set()
        
        with open(filepath, 'r', encoding='utf-8') as f:
            for line in f:
                syllable = line.strip()
                if syllable:
                    dataset.syllables.add(syllable)
        
        return dataset
    
    def __len__(self):
        return len(self.syllables)
    
    def __contains__(self, item):
        return item in self.syllables
```

</details>

<details>
<summary><code>nepali_syllabic_tokenizer.py</code></summary>

```python
"""
Nepali Syllabic Tokenizer V2 Implementation
Rule-based approach using Devanagari character sets and linguistic rules
"""

import re
import string
from typing import List
from syllable_dataset import SyllableDataset

class NepaliSyllabicTokenizer:
    """
    Rule-based syllabic tokenizer for Nepali language using Devanagari character sets

    Algorithm:
    1. Scan text left to right
    2. Each token starts with a vowel, consonant, or complex consonant
    3. Accumulate characters (vowel markers, halant, other markers) until the next
       vowel/consonant/complex consonant/space is reached
    """

    def __init__(self, normalize: bool = False, remove_non_devanagari: bool = True):
        """
        Initialize the tokenizer with Devanagari character sets

        Args:
            normalize: If True, normalizes characters before tokenization (default: False)
                      ई -> इ, श/ष -> स, ऊ -> उ, ी -> ि, ू -> ु
            remove_non_devanagari: If True, removes non-Devanagari characters (default: True)
        """
        self.normalize = normalize
        self.remove_non_devanagari = remove_non_devanagari

        # Normalization mapping
        self.normalization_map = {
            'ई': 'इ',
            'श': 'स',
            'ष': 'स',
            'ऊ': 'उ',
            'ी': 'ि',
            'ू': 'ु',
        }

    def _preprocess_text(self, text: str) -> str:
        """
        Preprocess text by replacing punctuations with spaces and normalizing spaces

        Args:
            text: Input text

        Returns:
            Preprocessed text with punctuations replaced by single spaces
            and multiple spaces collapsed to single spaces
        """
        # Replace all standard ASCII punctuation characters with single space
        for punct in string.punctuation:
            text = text.replace(punct, ' ')

        # Replace Nepali punctuation marks with single space
        nepali_punctuation = ['।', ',', '.', '?', '!', '-', '\'', '"']
        for punct in nepali_punctuation:
            text = text.replace(punct, ' ')

        # Replace multiple spaces with single space using regex
        text = re.sub(r'\s+', ' ', text)

        return text

    def _normalize_text(self, text: str) -> str:
        """
        Normalize text by replacing specific characters

        Args:
            text: Input text

        Returns:
            Normalized text
        """
        if not self.normalize:
            return text

        normalized = []
        for char in text:
            normalized.append(self.normalization_map.get(char, char))
        return ''.join(normalized)

    def tokenize(self, text: str) -> List[str]:
        """
        Tokenize Devanagari text into syllabic units.

        Rules:
        - Scan left to right
        - Each token starts with a vowel, consonant, or complex consonant
        - Accumulate characters (vowel markers, halant, other markers) until the next
          vowel/consonant/complex consonant/space is reached

        Args:
            text: Input text string in Devanagari

        Returns:
            List of tokens
        """
        if not text:
            return []

        # Step 0: Preprocess text - replace punctuations with spaces and normalize spaces
        text = self._preprocess_text(text)

        # Normalize text if enabled
        text = self._normalize_text(text)

        tokens = []
        i = 0

        while i < len(text):
            first_three_chars = text[i:i+3]
            char = text[i]

            # 1. space
            if char.isspace():
                tokens.append(" ")
                i += 1
                continue

            # 2. Check Complex consonant
            if first_three_chars in SyllableDataset.COMPLEX_CONSONANTS:
                tokens.append(first_three_chars)
                i += 3
                continue

            # 3. check if char is vowel or consonant
            elif char in SyllableDataset.VOWELS or char in SyllableDataset.CONSONANTS:
                    tokens.append(char)
                    i += 1
                    continue

            # 4. Handle vowel markers, halant, and other markers (attach to previous token)
            elif char in SyllableDataset.VOWEL_MARKERS + [SyllableDataset.HALANT] + SyllableDataset.OTHER_MARKERS:
                if tokens and not tokens[-1].isspace():
                    tokens[-1] += char
                else:
                    # Handle case where marker appears at beginning (shouldn't happen in valid text)
                    tokens.append(char)
                i += 1
                continue

            # Handle any other characters (numbers, invalid characters etc.)
            # todo: convert numbers to word in pre process? `SyllableDataset.NUMBERS?`
            if self.remove_non_devanagari:
                tokens.append(char)
            i += 1

        return list(tokens)

    def encode(self, text: str) -> List[str]:
        """Alias for tokenize method"""
        return self.tokenize(text)

    def decode(self, tokens: List[str]) -> str:
        """
        Decode tokens back to text

        Args:
            tokens: List of syllabic tokens

        Returns:
            Reconstructed text
        """
        return ''.join(tokens)

    def tokenize_batch(self, texts: List[str]) -> List[List[str]]:
        """
        Tokenize multiple texts

        Args:
            texts: List of input texts

        Returns:
            List of tokenized texts
        """
        return [self.tokenize(text) for text in texts]
```

</details>

## References

- Ghimire, R. R., Bal, B. K., Prasain, B., & Poudyal, P. (2023). [Pronunciation-Aware Syllable Tokenizer for Nepali Automatic Speech Recognition System](https://aclanthology.org/2023.icon-1.4/). ICON 2023.
- (ToDo) [Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Models](https://arxiv.org/abs/2512.03989v2)
- (ToDo) [How efficiently do popular tokenizers handle Devanagari Nepali? A fertility benchmark](https://www.himalayaai.org/blog/nepali-tokenizer-fertility-benchmark)