Mlflow & Evaluation
A team has deployed a customer service bot and is observing an increase in 'hallucinated' responses regarding product return policies. They need to quantitatively measure the accuracy of the retrieved context against the ground truth policy documents to identify if the issue lies with the retrieval quality or the generation model. Which evaluation method within the Databricks ecosystem provides the most direct empirical evidence for this specific gap?
To empirically diagnose hallucinations in RAG (Retrieval-Augmented Generation) systems, 'faithfulness' metrics are essential. These metrics evaluate whether the generated answer is strictly derived from the retrieved context. MLflow supports evaluation pipelines where frameworks like RAGAS can be used to compute faithfulness scores by comparing model output against the source chunks. Increasing temperature exacerbates hallucination; token counts measure cost not accuracy; and surveys provide qualitative feedback but do not isolate the retrieval vs. generation failure mode.