Prompt Evaluation & Expert Review — AI-500 Practice Question
A representative Microsoft Multi-Agent AI Solutions Expert (AI-500) exam question on Prompt Evaluation & Expert Review. Work through it below, then read why each option is right or wrong.
This question is part of the free Microsoft Multi-Agent AI Solutions Expert practice test.
Short answer
The correct answer is A. Use expert legal review to evaluate a sample of agent outputs across different contract types, assessing prompt clarity, tool selection, and reasoning quality; document findings and iterate on prompts based on expert feedback..
With limited labeled data, expert review (human-in-the-loop evaluation) is the appropriate governance control. Legal experts can assess prompt quality, tool selection, and reasoning across a diverse sample of contracts, providing feedback to improve end-to-end behavior without waiting for large labeled datasets. Collecting thousands of labeled contracts delays deployment indefinitely; relying on client feedback in production risks client harm; a rule-based template check loses the ability to evaluate nuanced legal reasoning.
The Question
A legal-services firm deploys a multi-agent contract review system but has only limited labeled data for training evaluation metrics. The firm must assess whether the system's prompts and tool invocations are producing legally sound recommendations despite the data scarcity. Which evaluation approach should the firm adopt to govern prompt quality and end-to-end behavior?
Why A is correct
With limited labeled data, expert review (human-in-the-loop evaluation) is the appropriate governance control. Legal experts can assess prompt quality, tool selection, and reasoning across a diverse sample of contracts, providing feedback to improve end-to-end behavior without waiting for large labeled datasets. Collecting thousands of labeled contracts delays deployment indefinitely; relying on client feedback in production risks client harm; a rule-based template check loses the ability to evaluate nuanced legal reasoning.
Why the other options are wrong
Option B does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (A) is the only one that fully meets every constraint stated in the question.
Option C does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (A) is the only one that fully meets every constraint stated in the question.
Option D does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (A) is the only one that fully meets every constraint stated in the question.
Key idea: Prompt Evaluation & Expert Review
With limited labeled data, expert review (human-in-the-loop evaluation) is the appropriate governance control. Legal experts can assess prompt quality, tool selection, and reasoning across a diverse sample of contracts, providing feedback to improve end-to-end behavior without waiting for large labeled datasets. Collecting thousands of labeled contracts delays deployment indefinitely; relying on client feedback in production risks client harm; a rule-based template check loses the ability to evaluate nuanced legal reasoning. On the AI-500 exam, questions in the "Evaluate, optimize, and monitor multi-agent solutions" domain test whether you can map a scenario's constraints to the right choice. Read the requirement carefully, eliminate options that violate any single constraint, and pick the one that satisfies all of them with the least operational overhead.
Ready to see how you'd score?
Take the free practice quiz and find out which Microsoft Multi-Agent AI Solutions Expert domains you need to focus on. No signup required.
Practice 5 similar questions
Same cert, same or adjacent domain. Use these after reviewing the explanation.
Related Microsoft Multi-Agent AI Solutions Expert Practice Questions
Official sources used
Verify program and exam details at the source before scheduling.