Batch Texts for an Embeddings API

Embedding APIs limit each request to a maximum number of inputs and a maximum total number of tokens. Batch a corpus to use as few requests as possible while respecting both limits, and dedupe identical texts so you don't pay twice.

Implement make_batches(texts, max_items, max_tokens, count_tokens=lambda t: len(t.split())):

• Dedupe exact duplicates, keeping the first occurrence order.
• Fill batches greedily in order: start a new batch when adding the next text would exceed max_items or max_tokens.
• A single text longer than max_tokens raises ValueError (it must be chunked first).
• Return (batches, index_map), where batches is a list of lists of unique texts and index_map[i] gives (batch_number, position) for every original text i, so results can be scattered back.