Mlflow & Retrieval Evaluation — Databricks GenAI Associate Practice Question
A representative Databricks Certified Generative AI Engineer Associate (Databricks GenAI Associate) exam question on Mlflow & Retrieval Evaluation. Work through it below, then read why each option is right or wrong.
This question is part of the free Databricks Certified Generative AI Engineer Associate practice test.
Short answer
The correct answer is B. Log retrieval precision and recall metrics separately, then use MLflow's LLM judge to evaluate whether the final summary answers the user's question accurately.
End-to-end RAG quality requires measuring both retrieval relevance and generation correctness. MLflow's LLM judge capability evaluates whether the final output meets user intent—not just embedding similarity. Retrieval metrics alone don't confirm the LLM used the context effectively. Latency and token counts are operational metrics, not quality measures. Embedding similarity doesn't validate that the retrieved content was actually useful for the downstream task. The correct approach combines retrieval evaluation (precision/recall) with LLM-judged generation quality to capture the full pipeline.
The Question
You are building a financial document retrieval system using Mosaic AI Vector Search. Your RAG pipeline retrieves quarterly earnings documents and passes them to an LLM for summarization. You want to measure whether the retrieved context actually helps the LLM produce accurate summaries. Which MLflow evaluation approach best captures this end-to-end quality?
Why B is correct
End-to-end RAG quality requires measuring both retrieval relevance and generation correctness. MLflow's LLM judge capability evaluates whether the final output meets user intent—not just embedding similarity. Retrieval metrics alone don't confirm the LLM used the context effectively. Latency and token counts are operational metrics, not quality measures. Embedding similarity doesn't validate that the retrieved content was actually useful for the downstream task. The correct approach combines retrieval evaluation (precision/recall) with LLM-judged generation quality to capture the full pipeline.
Why the other options are wrong
Option A does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (B) is the only one that fully meets every constraint stated in the question.
Option C does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (B) is the only one that fully meets every constraint stated in the question.
Option D does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (B) is the only one that fully meets every constraint stated in the question.
Key idea: Mlflow & Retrieval Evaluation
End-to-end RAG quality requires measuring both retrieval relevance and generation correctness. MLflow's LLM judge capability evaluates whether the final output meets user intent—not just embedding similarity. Retrieval metrics alone don't confirm the LLM used the context effectively. Latency and token counts are operational metrics, not quality measures. Embedding similarity doesn't validate that the retrieved content was actually useful for the downstream task. The correct approach combines retrieval evaluation (precision/recall) with LLM-judged generation quality to capture the full pipeline. On the Databricks GenAI Associate exam, questions in the "Evaluation and Monitoring" domain test whether you can map a scenario's constraints to the right choice. Read the requirement carefully, eliminate options that violate any single constraint, and pick the one that satisfies all of them with the least operational overhead.
Ready to see how you'd score?
Take the free practice quiz and find out which Databricks Certified Generative AI Engineer Associate domains you need to focus on. No signup required.
Practice 5 similar questions
Same cert, same or adjacent domain. Use these after reviewing the explanation.
Related Databricks Certified Generative AI Engineer Associate Practice Questions
Official sources used
Verify program and exam details at the source before scheduling.