Evals & Regression — CCDV-F Practice Question
A representative Claude Certified Developer - Foundations (CCDV-F) exam question on Evals & Regression. Work through it below, then read why each option is right or wrong.
Short answer
The correct answer is A. The eval set was not representative of production inputs and failure modes.
Hand-picked successes do not estimate production behavior. A representative regression set should include common cases, edge cases, and known failures.
The Question
A prompt change improves five hand-picked examples but user complaints rise after release. What test gap most likely caused this?
Why A is correct
Hand-picked successes do not estimate production behavior. A representative regression set should include common cases, edge cases, and known failures.
Why the other options are wrong
Option B does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (A) is the only one that fully meets every constraint stated in the question.
Option C does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (A) is the only one that fully meets every constraint stated in the question.
Option D does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (A) is the only one that fully meets every constraint stated in the question.
Key idea: Evals & Regression
Hand-picked successes do not estimate production behavior. A representative regression set should include common cases, edge cases, and known failures. On the CCDV-F exam, questions in the "Eval, Testing, and Debugging" domain test whether you can map a scenario's constraints to the right choice. Read the requirement carefully, eliminate options that violate any single constraint, and pick the one that satisfies all of them with the least operational overhead.
Ready to see how you'd score?
Take the free practice quiz and find out which Claude Certified Developer - Foundations domains you need to focus on. No signup required.
Practice 5 similar questions
Same cert, same or adjacent domain. Use these after reviewing the explanation.