Byte-level tokenizers (GPT-2, Llama 3) can split one character across several tokens: é is the two bytes C3 A9, and an emoji is four. If you decode each streamed token on its own, users see � garbage.
Implement a class StreamDecoder(vocab), where vocab maps token ids to bytes:
feed(token_id) -> str: return only the text that is now complete, holding back any trailing partial characterflush() -> str: at the end of the stream, return whatever is left, with any incomplete bytes replaced by "�"decode(ids) -> str: decode a whole list at once (an invalid sequence also becomes "�")Feeding all tokens and then flushing must give exactly the same string as decode.