Once a BPE tokenizer is trained, its merge list (ranked pairs) defines how to encode any new word. Starting from single characters, repeatedly find the adjacent pair with the lowest rank (earliest learned) and merge every occurrence of it, scanning left to right without overlaps. Stop when no adjacent pair is in the merge list.
Implement:
encode_word(word, merges): merges is a list of (a, b) pairs in rank order. Return the list of tokens.encode(text, merges, vocab): split text on single spaces, encode each word, and map tokens to ids with vocab. Tokens missing from the vocabulary map to vocab["<unk>"].
Between words, insert the id of the " " token.decode(ids, vocab): map ids back to tokens and concatenate them.