Before tuning prompts, check that your retriever finds the right documents. Three standard metrics, for one query with a ranked list retrieved and a set of relevant document ids:
Implement recall_at_k, reciprocal_rank and ndcg_at_k, each taking (retrieved, relevant) (plus k where named). Then implement evaluate(runs, k), where runs is a list of (retrieved, relevant) pairs, returning a dict with the mean "recall@k", "mrr" and "ndcg@k" across queries.
If relevant is empty, recall and nDCG are 0.