The same sentence can cost 2–3× more tokens in Hindi or Thai than in English, which means higher API bills and less effective context. Measure it.
tokenize is any function from a string to a list of tokens. Implement:
fertility(texts, tokenize): total tokens divided by total whitespace-separated wordschars_per_token(texts, tokenize): total characters divided by total tokenslanguage_report(corpus, tokenize, baseline="en"): corpus maps language codes to lists of parallel texts. Return {lang: {"fertility": ..., "cost_vs_baseline": ...}}, where the cost ratio is that language's total tokens divided by the baseline's.