Word Error Rate and Character Error Rate are the two most widely used metrics for evaluating language generation tasks - from transcription to machine translation. Here is exactly how they are calculated on Zindi, and how adjusting the weight_factor changes how short and long transcripts impact your final score.

WER and CER: what they measure

Both metrics share the same engine - Levenshtein distance (minimum edit distance) - but operate at different granularities. For each sample, the algorithm counts the minimum number of insertions, deletions, and substitutions needed to turn the hypothesis (your model's output) into the reference (the correct transcript).

Edit distance (applies to both metrics)

Edits = Substitutions (S) + Deletions (D) + Insertions (I)

WER = Edits / (number of words in reference) .

CER = Edits / (number of characters in reference)

What "weighted" means - The Sample Weight Factor

A naive per-sample average lets a model gain ground by perfecting short, easy utterances. The weighted variants of WER and CER prevent this by scaling each sample's contribution by the square root of its reference length - so longer transcripts carry more weight, but the relationship is sublinear.

A sample's weight (W_i) is calculated dynamically by raising its reference sequence length (L_i) to the power of the weight_factor:W_i = L_i^{weight_factor} where L_i represents:

The number of words in the reference string for WER.

The number of characters in the reference string for CER.

W_i = L_i^{0.5} = \sqrt{L_i}

The final global score for either metric is calculated as the sum of all weighted edits divided by the total sum of all sample weights across the dataset:

Global Score = (sum of weighted edits) ÷ (sum of sample weights)

where each sample's weighted edits = Editsᵢ × √Lᵢ

How the Weight Factor Shifts the Balance:

1. weight_factor = 1.0 (Standard Global Score):

The weight is exactly equal to the length (W_i = L_i^1 = L_i). This yields traditional unweighted global WER/CER. Because longer sentences have more words/characters, they naturally carry a massive proportional weight in the final score.

2. weight_factor = 0.5 (Sublinear / Dampened Weight):

The weight scales by the square root of the length (W_i = L_i^{0.5} = \sqrt{L_i}). This mathematical dampening prevents a few exceptionally long transcripts from completely dominating the metric, bringing the evaluation closer to treating long and short utterances equally.

What to watch out for

• Predict every ID. Missing predictions are penalised as incorrect - they're not skipped. The edit distance is computed as if your model output an empty string.
• Conversational tags are stripped automatically. Role markers like user, assistant, <|user|>, <|assistant|> are removed before scoring - no need to clean LLM-format outputs yourself.
• On weighted variants, a single long empty prediction can dominate your global score far more than on the unweighted version. Always predict something - even a rough output - rather than leaving a field blank.

Let’s Go Through a Step-by-Step Calculation Example (WER)

Consider an evaluation dataset containing only two samples to see exactly how the weight_factor alters your score.

Sample 1: Long sentence (L_1 = 10 words)

• Reference: "the quick brown fox jumps over the lazy dog today"
• Hypothesis: "the quick brown fox jumps over the lazy dog" (1 word deleted: missed "today")

Sample 2: Short sentence (L_2 = 2 words)

• Reference: "go home"
• Hypothesis: "go" (1 word deleted: missed "home")

How the Weights are Calculated

Before looking at the final score, see how the two different weight factors evaluate these sentences:

At weight_factor = 1.0: Weights match the word counts perfectly.

Sample 1 Weight = 10^1 = 10

Sample 2 Weight = 2^1 = 2

Total Dataset Weight = 10 + 2 = 12

At weight_factor = 0.5: Weights are dampened using the square root.

Sample 1 Weight = \sqrt{10} \approx 3.1622

Sample 2 Weight = \sqrt{2} \approx 1.4142

Total Dataset Weight = 3.1622 + 1.4142 = 4.5764

Why This Matters For Your Leaderboard Standing

Imagine you improve your model so that it perfectly fixes the long sentence, but it still misses that single word in the short sentence (Sample 1 Edits = 0, Sample 2 Edits = 1).

Scenario A: Traditional Global WER (weight_factor = 1.0)

Sample 1 Weighted Edits = 0 x 10 = 0

Sample 2 Weighted Edits = 1 x 2 = 2

Global WER = (0 + 2) / 12 = {0.1667}

The Takeaway: Your score looks amazing (16.67% error) because the flawless long sentence completely dilutes your mistake on the short one.

Scenario B: Sublinear Weighted WER (weight_factor = 0.5)

Sample 1 Weighted Edits = 0 x 3.1622 = 0

Sample 2 Weighted Edits = 1 x 1.4142 = 1.4142

Global WER = (0 + 1.4142) / 4.5764 = {0.3090}

The Takeaway: The error on the short sentence now penalizes your total score nearly twice as hard (30.90% vs 16.67%) because the long sentence's structural weight has been suppressed.

When the weight factor shrinks, every sentence - regardless of size - demands your model's full attention!