Provider prompt caches (and vLLM's prefix cache) only reuse an exact prefix. Putting a timestamp or the user's name at the top of the prompt defeats caching for everything after it.
A prompt is a list of components (name, tokens, static), where static means identical across requests. Implement:
cacheable_tokens(components): the token count of the leading run of static components (stop at the first dynamic one)reorder(components): move every static component before every dynamic one, keeping the relative order within each groupcost(components, n_requests, price, cached_discount=0.9): the total input cost over n_requests identical-layout requests. The first request pays full price for everything; later requests pay price × (1 - cached_discount) for the cacheable tokens and full price for the rest. price is per token.