SentencePiece's Unigram tokenizer (used by T5, ALBERT and XLNet) gives each vocabulary piece a log-probability, and tokenizes a string by choosing the segmentation with the highest total log-probability. Finding it is a shortest-path problem solved with the Viterbi algorithm.
Unlock this and every premium problem across all tracks — plus hidden-test grading, advanced AI engineering modules and new content every week.
Start 7-day free trial