Compress Prompts by Self-Information

LLMLingua-style compression drops the tokens a small language model finds predictable. A token's self-information is $-\log p(\text{token} \mid \text{previous tokens})$: high means surprising and informative, low means you can probably infer it.

Implement compress(tokens, logprobs, budget, protected=()):

• Always keep the indices in protected (for example the user's question).
• Fill the rest of the budget with the unprotected tokens of highest self-information (ties: earlier index).
• Return the kept tokens in their original order. If the budget is smaller than the protected set, keep just the protected tokens.

Also implement compression_ratio(original_len, kept_len): original_len / kept_len.