Fixed-size chunking cuts sentences in half. LangChain's popular recursive splitter tries the most natural boundaries first: split on paragraphs, and if a piece is still too big, split it on lines, then sentences, then words, then characters.
Implement recursive_split(text, chunk_size, separators=("\n\n", "\n", ". ", " ", "")):
len(text) <= chunk_size, return [text] (or [] if it's empty or only whitespace)."" means individual characters). Keep the separator at the end of each piece except the last, so no text is lost.<= chunk_size.chunk_size is split recursively with the remaining separators.Joining all chunks must reproduce the original text exactly.