Sentence-Aware Chunking with Overlap

Chunk documents at sentence boundaries with a token budget and a little overlap, so a fact split across a boundary still appears whole in one chunk.

Implement chunk_sentences(text, max_tokens, overlap_sentences=1, count_tokens=lambda s: len(s.split())):

• Split into sentences after ., ! or ? followed by whitespace. Strip each sentence and drop empty ones.
• Build chunks greedily: add sentences while the chunk's total tokens stay <= max_tokens.
• Each new chunk starts with the last overlap_sentences sentences of the previous chunk (if they fit with the next sentence; drop overlap sentences from the front until they do).
• A single sentence longer than max_tokens becomes its own chunk, unsplit.
• Join sentences in a chunk with a single space. Never emit a chunk that consists only of overlap.