Offline Evaluation & Online Evaluation — AI-300 Practice Question
A representative Microsoft Machine Learning Operations Engineer Associate (AI-300) exam question on Offline Evaluation & Online Evaluation. Work through it below, then read why each option is right or wrong.
This question is part of the free Microsoft Machine Learning Operations Engineer Associate practice test.
Short answer
The correct answer is C. Implement offline batch evaluation on historical data weekly, combined with online monitoring of latency and user feedback metrics in production.
C is correct. Offline evaluation on historical data enables cost-effective, controlled assessment of model quality without impacting production users, while online monitoring captures real-world performance, latency, and user feedback. This dual approach aligns with Azure ML best practices for model monitoring. A risks poor user experience during experimentation. D misses production drift signals. B is computationally infeasible and violates the 200ms latency requirement. The correct answer balances safety, cost, and responsiveness.
The Question
You are designing an evaluation strategy for a generative AI application that provides customer service recommendations. The model is deployed in production and must serve responses within 200ms. You need to assess both model quality and user experience continuously. Which combination of evaluation approaches is most appropriate for this scenario?
Why C is correct
C is correct. Offline evaluation on historical data enables cost-effective, controlled assessment of model quality without impacting production users, while online monitoring captures real-world performance, latency, and user feedback. This dual approach aligns with Azure ML best practices for model monitoring. A risks poor user experience during experimentation. D misses production drift signals. B is computationally infeasible and violates the 200ms latency requirement. The correct answer balances safety, cost, and responsiveness.
Why the other options are wrong
Option A does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (C) is the only one that fully meets every constraint stated in the question.
Option B does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (C) is the only one that fully meets every constraint stated in the question.
Option D does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (C) is the only one that fully meets every constraint stated in the question.
Key idea: Offline Evaluation & Online Evaluation
C is correct. Offline evaluation on historical data enables cost-effective, controlled assessment of model quality without impacting production users, while online monitoring captures real-world performance, latency, and user feedback. This dual approach aligns with Azure ML best practices for model monitoring. A risks poor user experience during experimentation. D misses production drift signals. B is computationally infeasible and violates the 200ms latency requirement. The correct answer balances safety, cost, and responsiveness. On the AI-300 exam, questions in the "Implement quality assurance and observability for generative AI" domain test whether you can map a scenario's constraints to the right choice. Read the requirement carefully, eliminate options that violate any single constraint, and pick the one that satisfies all of them with the least operational overhead.
Ready to see how you'd score?
Take the free practice quiz and find out which Microsoft Machine Learning Operations Engineer Associate domains you need to focus on. No signup required.
Practice 5 similar questions
Same cert, same or adjacent domain. Use these after reviewing the explanation.
Related Microsoft Machine Learning Operations Engineer Associate Practice Questions
Official sources used
Verify program and exam details at the source before scheduling.