RetrievalEvaluator Class

Initialize an evaluator configured for a specific Azure OpenAI model.

Usage

   chat_eval = RetrievalEvaluator(model_config)
   conversation = {
       "messages": [
           {"role": "user", "content": "What is the value of 2 + 2?"},
           {
               "role": "assistant", "content": "2 + 2 = 4",
               "context": "From 'math_doc.md': Information about additions: 1 + 2 = 3, 2 + 2 = 4"
           }
       ]
   }
   result = chat_eval(conversation=conversation)

Output format

   {
       "gpt_retrieval": 3.0,
       "retrieval": 3.0,
       "evaluation_per_turn": {
           "gpt_retrieval": [1.0, 2.0, 3.0],
           "retrieval": [1.0, 2.0, 3.0],
           "retrieval_reason": ["<reasoning for score 1>", "<reasoning for score 2>", "<reasoning for score 3>"]
       }
   }

Note: To align with our support of a diverse set of models, a key without the gpt_ prefix has been added. To maintain backwards compatibility, the old key with the gpt_ prefix is still be present in the output; however, it is recommended to use the new key moving forward as the old key will be deprecated in the future.