The QA Relevance LLM validator compares the similarity between the input prompt and LLM-generated output. After comparison, the validator generates a confidence score representing the contextual relevance of the response. This validator ensures that LLM outputs are relevant to user queries, fostering a good user experience. The validator measures relevance in three steps:
1. The scanner converts both the prompt and the output into vector embeddings.
2. It calculates the cosine similarity between these embeddings.
3. This similarity score is compared against a predefined threshold to determine relevance.