Model Evaluation & Bedrock Evaluations — AWS GenAI Dev Practice Question
A representative AWS Generative AI Developer (AWS GenAI Dev) exam question on Model Evaluation & Bedrock Evaluations. Work through it below, then read why each option is right or wrong.
Short answer
The correct answer is B. Use Amazon Bedrock Evaluations with automatic evaluation, configuring both faithfulness (to measure hallucination against source documents) and answer relevance metrics, then analyze the correlation between retrieval results and generation quality.
Amazon Bedrock Evaluations supports automatic evaluation with built-in metrics specifically designed for RAG systems. Faithfulness measures whether generated answers are grounded in the retrieved source documents (identifying contradictions), while answer relevance measures the helpfulness and format of responses. By analyzing both metrics, the developer can distinguish between retrieval failures (wrong documents retrieved) and generation failures (correct documents but poor synthesis).
The Question
A developer has built a RAG-based question-answering system using Amazon Bedrock Knowledge Bases. Users are reporting that the system sometimes generates answers that contradict the source documents, and other times provides correct information but in an unhelpful format. The developer needs to systematically evaluate the system's performance and identify which component (retrieval or generation) is causing each type of issue. Which evaluation approach should the developer use?
Why B is correct
Amazon Bedrock Evaluations supports automatic evaluation with built-in metrics specifically designed for RAG systems. Faithfulness measures whether generated answers are grounded in the retrieved source documents (identifying contradictions), while answer relevance measures the helpfulness and format of responses. By analyzing both metrics, the developer can distinguish between retrieval failures (wrong documents retrieved) and generation failures (correct documents but poor synthesis).
Why the other options are wrong
Human evaluation provides subjective quality ratings but doesn't systematically separate retrieval vs. generation issues. A 1-5 overall quality scale lacks the granularity needed to diagnose whether contradictions stem from bad retrieval or bad generation. Human evaluation is also slower and more expensive for systematic analysis.
While a custom LLM-as-judge approach could work, it requires significant development effort to build and maintain. Amazon Bedrock Evaluations provides this capability natively with validated metrics, making a custom pipeline unnecessary overhead.
A/B testing different models addresses model selection but doesn't help diagnose whether issues are in retrieval or generation. Both models would use the same Knowledge Base and retrieval pipeline, so retrieval-caused contradictions would persist across both models.
Key idea: Model Evaluation & Bedrock Evaluations
Why B is correct: Amazon Bedrock Evaluations supports automatic evaluation with built-in metrics specifically designed for RAG systems. Faithfulness measures whether generated answers are grounded in the retrieved source documents (identifying contradictions), while answer relevance measures the helpfulness and format of responses. By analyzing both metrics, the developer can distinguish between retrieval failures (wrong documents retrieved) and generation failures (correct documents but poor synthesis). Why A is wrong: Human evaluation provides subjective quality ratings but doesn't systematically separate retrieval vs. generation issues. A 1-5 overall quality scale lacks the granularity needed to diagnose whether contradictions stem from bad retrieval or bad generation. Human evaluation is also slower and more expensive for systematic analysis. Why C is wrong: While a custom LLM-as-judge approach could work, it requires significant development effort to build and maintain. Amazon Bedrock Evaluations provides this capability natively with validated metrics, making a custom pipeline unnecessary overhead. Why D is wrong: A/B testing different models addresses model selection but doesn't help diagnose whether issues are in retrieval or generation. Both models would use the same Knowledge Base and retrieval pipeline, so retrieval-caused contradictions would persist across both models. On the AWS GenAI Dev exam, questions in the "Testing, Validation, and Troubleshooting" domain test whether you can map a scenario's constraints to the right choice. Read the requirement carefully, eliminate options that violate any single constraint, and pick the one that satisfies all of them with the least operational overhead.
Ready to see how you'd score?
Take the free practice quiz and find out which AWS Generative AI Developer - Professional domains you need to focus on. No signup required.
Practice 5 similar questions
Same cert, same or adjacent domain. Use these after reviewing the explanation.