AI Engineer Interview Practice Test: 24 Production Scenarios

Production judgment more than definitions. The questions that separate builders from tinkerers are open-ended: how you would build a RAG system and where it fails in production, when you would fine-tune instead of using RAG, how you evaluate outputs with no single right answer, and how you design an agent that makes several tool calls without looping. This test turns those situations into 24 scored scenarios. The 12 open-ended questions, each with a guide to what a strong answer covers, are free at aijobprep.app/questions/ai-engineer.html.

Walk the pipeline in order: ingestion, chunking, embeddings, a vector store with metadata filters, hybrid retrieval, reranking, prompt assembly, and generation with citations. Then lead with the production failures: bad retrieval is the top cause, followed by stale indexes, context stuffing, and no eval set. Candidates who name the failure modes first signal real experience.

Fine-tuning teaches behavior and format. RAG supplies knowledge. Fine-tune for consistent output structure at scale, style matching, moving a narrow task to a smaller and cheaper model, or classification with plenty of labels. Do not fine-tune to inject facts that change. Say that you need an eval set before any fine-tune, or you cannot tell whether it worked.

Build a golden set from real usage, score it with a rubric, and use a model grader whose biases you have measured against human labels. Prefer pairwise comparison over absolute scores, and run the eval in CI so a prompt change cannot degrade quality silently. Name one metric you would track every week. Several questions in this test cover the trap where the grader and the experts disagree.

Ground answers in retrieved sources with citations, constrain the model to say it does not know when the answer is not in context, verify high-stakes claims after generation, and route low-confidence cases to a person. For injection, the security boundary lives where the side effect happens: least-privilege tools, server-side argument checks, and confirmation for consequential actions. Instructions to the model are advisory, never a control.

Decompose cost per request by feature before you touch anything, because a 10x bill on flat traffic means cost per request exploded. The usual drivers are context growth, retry loops, and output length. Then apply the levers in order of impact: prompt caching for repeated context, routing easy queries to a smaller model, trimming the system prompt and retrieved context, batching where latency allows, and semantic caching. Re-run the eval set after each step.

The questions are the AI Engineer quiz from aijobprep.app, built from a corpus of 198 AI company job posts and a 29-question member bank. Each one is a scenario with a production constraint, and the explanation names the habit it tests: decompose before you fix, contain the agent in the harness, and audit what an eval measures before you trust it.