RelevanceTruthAndCompletenessEvaluator Class

Reference

Definition

Namespace:: Microsoft.Extensions.AI.Evaluation.Quality

Assembly:: Microsoft.Extensions.AI.Evaluation.Quality.dll

Package:: Microsoft.Extensions.AI.Evaluation.Quality v9.4.0-preview.1.25207.5

Important

Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.

An IEvaluator that evaluates the 'Relevance', 'Truth' and 'Completeness' of a response produced by an AI model.

public ref class RelevanceTruthAndCompletenessEvaluator sealed : Microsoft::Extensions::AI::Evaluation::Quality::ChatConversationEvaluator

public sealed class RelevanceTruthAndCompletenessEvaluator : Microsoft.Extensions.AI.Evaluation.Quality.ChatConversationEvaluator

type RelevanceTruthAndCompletenessEvaluator = class
    inherit ChatConversationEvaluator

Public NotInheritable Class RelevanceTruthAndCompletenessEvaluator
Inherits ChatConversationEvaluator

Inheritance: Object

ChatConversationEvaluator
RelevanceTruthAndCompletenessEvaluator

Remarks

RelevanceTruthAndCompletenessEvaluator returns three NumericMetrics that contain scores for 'Relevance', 'Truth' and 'Completeness' respectively. Each score is a number between 1 and 5, with 1 indicating a poor score, and 5 indicating an excellent score. Each returned score is also accompanied by a Reason that provides an explanation for the score.

Note: RelevanceTruthAndCompletenessEvaluator is an AI-based evaluator that uses an AI model to perform its evaluation. While the prompt that this evaluator uses to perform its evaluation is designed to be model-agnostic, the performance of this prompt (and the resulting evaluation) can vary depending on the model used, and can be especially poor when a smaller / local model is used.

The prompt that RelevanceTruthAndCompletenessEvaluator uses has been tested against (and tuned to work well with) the following models. So, using this evaluator with a model from the following list is likely to produce the best results. (The model to be used can be configured via ChatClient.)

GPT-4o

Constructors

RelevanceTruthAndCompletenessEvaluator()

Properties

CompletenessMetricName	Gets the Name of the NumericMetric returned by RelevanceTruthAndCompletenessEvaluator for 'Completeness'.
EvaluationMetricNames	Gets the Names of the EvaluationMetrics produced by this IEvaluator.
IgnoresHistory	Gets a value indicating whether this IEvaluator considers the entire conversation history (in addition to the request and response being evaluated) as part of the evaluation it performs. (Inherited from ChatConversationEvaluator)
RelevanceMetricName	Gets the Name of the NumericMetric returned by RelevanceTruthAndCompletenessEvaluator for 'Relevance'.
SystemPrompt	Gets the system prompt that this IEvaluator uses when performing evaluations. (Inherited from ChatConversationEvaluator)
TruthMetricName	Gets the Name of the NumericMetric returned by RelevanceTruthAndCompletenessEvaluator for 'Truth'.

Methods

CanRenderAsync(ChatMessage, Int32, ChatConfiguration, CancellationToken)	Determines if there is sufficient `tokenBudget` remaining to render the supplied `message` as part of the evaluation prompt that this IEvaluator uses. (Inherited from ChatConversationEvaluator)
EvaluateAsync(IEnumerable<ChatMessage>, ChatResponse, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics. (Inherited from ChatConversationEvaluator)
InitializeResult()	Returns an EvaluationResult that includes default values for all the EvaluationMetrics supported by this IEvaluator. (Inherited from ChatConversationEvaluator)
PerformEvaluationAsync(ChatConfiguration, IList<ChatMessage>, EvaluationResult, CancellationToken)	Invokes the supplied ChatClient with the supplied `evaluationMessages` to perform the evaluation, and includes the results as one or more EvaluationMetrics in the supplied `result`. (Inherited from ChatConversationEvaluator)
RenderAsync(ChatMessage, CancellationToken)	Renders the supplied `message` to a string that can be included as part of the evaluation prompt that this IEvaluator uses. (Inherited from ChatConversationEvaluator)
RenderAsync(ChatResponse, CancellationToken)	Renders the supplied `response` to a string that can be included as part of the evaluation prompt that this IEvaluator uses. (Inherited from ChatConversationEvaluator)
RenderEvaluationPromptAsync(ChatMessage, ChatResponse, IEnumerable<ChatMessage>, IEnumerable<EvaluationContext>, CancellationToken)	Renders the information present in the supplied parameters into a prompt that this IEvaluator uses to perform the evaluation. (Inherited from ChatConversationEvaluator)

Extension Methods

EvaluateAsync(IEvaluator, ChatMessage, ChatMessage, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics.
EvaluateAsync(IEvaluator, ChatMessage, ChatResponse, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics.
EvaluateAsync(IEvaluator, ChatMessage, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics.
EvaluateAsync(IEvaluator, ChatResponse, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics.
EvaluateAsync(IEvaluator, String, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics.
EvaluateAsync(IEvaluator, String, String, ChatConfiguration, IEnumerable<EvaluationContext>, CancellationToken)	Evaluates the supplied `modelResponse` and returns an EvaluationResult containing one or more EvaluationMetrics.

Applies to

Share via