Beyond English: language, script and region

A pilot that works in English tells you almost nothing about Hindi, Arabic or Bahasa Indonesia. Tokenisers split non-Latin scripts into more pieces, so the same message can cost two to several times as many tokens, fill the context window faster and hit rate limits sooner; the ratio depends on the tokeniser, so you measure it per model. Quality differs by language too, usually more for generation than comprehension, and real users mix languages and scripts in one sentence. Right-to-left scripts and locale conventions (dates, numbers, currency, names) break UIs and downstream systems in ways that have nothing to do with model accuracy. You choose, per language, between a model that works natively and a pipeline that translates to a pivot language, and you consider regional or sovereign providers where language fit or residency demands it. Then you evaluate each language separately, with native speakers, before you claim anything.

• Measure token inflation per model and per language. Do not extrapolate from English.
• Code-switching is the norm, not the edge case, in many markets.
• Let the model produce structured values; let code format them per locale.
• Native vs translate-pivot is a per-language, per-task decision with cost, quality and audit trade-offs.
• Every language gets its own eval slice and a native reviewer, or it is not launched.