Hardest Claude Certified Developer - Foundations questions in the last 90 days

Messages API calls are stateless with respect to conversation history. The application owns state and must include the relevant prior turns in each request.

HTTP 529 is documented by Anthropic as 'API overloaded'—a server-side infrastructure error signaling the API cannot accept new requests at that moment. This is unambiguously an API-layer failure, not a model-quality failure; no prompt or output evaluation is needed to diagnose it. Re-running prompts synchronously to compare quality scores tests model behavior, not infrastructure availability, and cannot resolve a 529. Increasing max_tokens addresses response length, which has no bearing on overload errors. Switching model variants conflates model availability with server capacity; 529 is not model-specific.

Claude Code's .claude/settings.json supports a 'permissions' block where operators can explicitly allow or deny individual tools and require human approval for sensitive actions. This is the authoritative mechanism for controlling tool invocation behavior at the repository level. A CLAUDE.md description alone provides context but does not enforce tool restrictions—Claude may still invoke disallowed tools. Switching models via ANTHROPIC_MODEL addresses cost, not permission enforcement. A pre-commit hook operates on git commits, not on Claude Code tool execution, and cannot selectively restrict specific tools mid-session.

A rigorous evaluation compares the two configuration layers—CLAUDE.md instructions alone versus CLAUDE.md combined with .claude/settings.json permission enforcement—across repeated sessions, measuring objective violation rates. This surfaces whether natural-language instructions reliably produce compliant behavior or whether enforceable permission rules are necessary. A single manual inspection lacks statistical power and cannot reveal intermittent failures. Grepping stdout for an encoding string checks only that the string appeared, not that the correct tool behavior occurred. Self-reporting by Claude is unreliable as a compliance metric because the model cannot objectively audit its own tool calls.

The sequence and control logic are known, so deterministic orchestration is simpler and safer. Claude can handle the ambiguous extraction while code enforces schema and write rules.

Tool descriptions and schemas are part of the model's selection interface. Clear, mutually distinct contracts directly address systematic misselection.

A tool-use stop indicates that the model is waiting for a tool result. The client must execute the authorized tool, return a correctly formed result block, and continue the loop.

Distinguishing API failures from model failures requires capturing different signals at different layers. The transcript—the complete messages array including all tool calls, intermediate results, and returned responses—records what the model did. The outcome—the final state in the environment, such as whether a database record was created or a classification matched the label—records whether the task succeeded. HTTP status codes and stop reasons surface infrastructure problems: a 529 overloaded error, a 429 rate-limit error, or a network timeout is an API failure regardless of what the model would have said. A response with a 200 status and a valid stop reason that nonetheless answers incorrectly is a model failure. Keeping these signals separate allows each failure type to be routed to the correct remediation workflow. Treating every wrong answer as a model failure and every SDK exception as an API failure conflates cases where an API error caused a degraded but non-exception response, and misses cases where a malformed model output triggers a downstream exception. The classification is too coarse to drive targeted remediation. Running cases synchronously and logging only final text discards the intermediate signals—tool calls, stop reasons, status codes—needed to diagnose root cause. Human post-hoc review cannot recover information that was never recorded. Prompt caching is a latency and cost optimization. Cache state reflects whether a prompt prefix was reused, not whether the API layer or the model layer caused a failure. Using cache hits and misses as a proxy for failure classification produces a categorization with no causal relationship to the actual failure modes.

Credentials must never appear in prompts. Prompt content can be logged by the application layer, cached by the API, echoed back in model outputs, or exposed through prompt-injection attacks—any of these paths would leak the password. A secrets manager keeps credentials entirely outside the conversation, injecting them only at the application layer when a specific, authorized action is about to execute. Separately, permanent deletion is a consequential and difficult-to-reverse action. Requiring an explicit human-approval step before each deletion—with the approver's identity and a timestamp recorded—creates the auditable, reversible-within-90-days trail the regulation demands. These two controls together satisfy both requirements. Hashing the password before embedding it in the system prompt does not protect the credential: the hash is still present in prompt logs and caches, and a hash of a known password can be used to verify guesses or, for weak passwords, reversed. The plaintext is also still needed by the database driver, so the application must store it somewhere anyway. Rotating the password daily reduces the window of exposure after a leak but does not prevent the leak itself. The password is still written into the prompt on every request and is therefore still exposed to every logging and caching pathway. Removing delete capability entirely eliminates the credential-in-prompt risk but also eliminates a legitimate operational function. The correct approach is to govern the capability with proper secrets handling and human approval, not to abandon it.

Hand-picked successes do not estimate production behavior. A representative regression set should include common cases, edge cases, and known failures.