AnthropicEvaluation, Testing, and Optimization

Evals & Release Gate — CCAR-P Practice Question

A representative Claude Certified Architect - Professional (CCAR-P) exam question on Evals & Release Gate. Work through it below, then read why each option is right or wrong.

This question is part of the free Claude Certified Architect - Professional practice test.

Short answer

The correct answer is A. Not until the risk-specific threshold is met and the violation increase is understood.

Averages must not hide critical regressions. Risk-weighted gates should block a release that worsens an explicitly protected safety outcome.

The Question

An assistant's overall success rate rises from 84% to 89%, but policy-violation failures double from 0.2% to 0.4%. Should the release proceed?

ANot until the risk-specific threshold is met and the violation increase is understoodCorrect

BYes, as long as the new model costs less per request than the current one

CDecide from the outcome of one live demo shown to a single senior stakeholder

DYes, because the average success rate improved by five points across all tasks

Why A is correct

Averages must not hide critical regressions. Risk-weighted gates should block a release that worsens an explicitly protected safety outcome.

Why the other options are wrong

Option B: Yes, as long as the new model costs less per request than the current one

Option B does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (A) is the only one that fully meets every constraint stated in the question.

Option C: Decide from the outcome of one live demo shown to a single senior stakeholder

Option C does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (A) is the only one that fully meets every constraint stated in the question.

Option D: Yes, because the average success rate improved by five points across all tasks

Option D does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (A) is the only one that fully meets every constraint stated in the question.

Key idea: Evals & Release Gate

Averages must not hide critical regressions. Risk-weighted gates should block a release that worsens an explicitly protected safety outcome. On the CCAR-P exam, questions in the "Evaluation, Testing, and Optimization" domain test whether you can map a scenario's constraints to the right choice. Read the requirement carefully, eliminate options that violate any single constraint, and pick the one that satisfies all of them with the least operational overhead.

Ready to see how you'd score?

Take the free practice quiz and find out which Claude Certified Architect - Professional domains you need to focus on. No signup required.

Take the free quiz See all 160+ practice questions

Practice 5 similar questions

Same cert, same or adjacent domain. Use these after reviewing the explanation.

Take the quiz →

1Discovery & Requirements — CCAR-P Practice QuestionSolution Design and Architecture 2Model Routing & Governance — CCAR-P Practice QuestionClaude Models, Prompting, and Context Engineering 3RAG & Evaluation — CCAR-P Practice QuestionIntegration 4Governance & Data Retention — CCAR-P Practice QuestionGovernance, Safety, and Risk Management 5Stakeholders & Tradeoffs — CCAR-P Practice QuestionStakeholder Communication and Lifecycle Management

Related Claude Certified Architect - Professional Practice Questions

Discovery & Requirements — CCAR-P Practice Question→Model Routing & Governance — CCAR-P Practice Question→RAG & Evaluation — CCAR-P Practice Question→

Official sources used

Verify program and exam details at the source before scheduling.

AnthropicDemystifying Evals For AI Agents