Model Evaluation Metrics — Accuracy, Precision, Recall, F1
The AWS AI Practitioner (AIF-C01) exam will hand you a classification scenario and ask which metric to optimize. Accuracy is rarely the right answer. This page walks through how to decide between precision, recall, F1, and AUC-ROC.
Short answer
The correct answer is C. Recall.
Recall, also called sensitivity or true positive rate, measures the fraction of actual positive cases that the model correctly identifies. It is defined as true positives divided by the sum of true positives and false negatives. In this medical screening scenario, a false negative is a patient who truly has disease but is told the X-ray looks clean — the exact outcome the clinical team says is the most costly. Maximizing recall minimizes false negatives, which is what you want when missing a positive case causes real harm. A secondary human review by a radiologist acts as a safety net for false positives, so a slightly lower precision is an acceptable trade-off. Accuracy is a poor choice here because medical datasets are usually imbalanced; a model that predicts negative for every image can score 98 percent accuracy while missing every sick patient. Precision would be the right answer in the opposite setup — for example, an automated action that cannot be reviewed — but not when a human reviewer already catches false positives. Mean squared error is a regression metric and does not apply to a binary classification output.
The Question
A hospital is building a binary classification model that screens chest X-rays and flags images that may show early-stage disease. Every flagged image is then reviewed by a radiologist. The clinical team says that missing a true case of disease is far more costly than sending a healthy patient for a second review. Which evaluation metric should the ML team prioritize when tuning this model?
Why C is correct
Recall, also called sensitivity or true positive rate, measures the fraction of actual positive cases that the model correctly identifies. It is defined as true positives divided by the sum of true positives and false negatives. In this medical screening scenario, a false negative is a patient who truly has disease but is told the X-ray looks clean — the exact outcome the clinical team says is the most costly. Maximizing recall minimizes false negatives, which is what you want when missing a positive case causes real harm. A secondary human review by a radiologist acts as a safety net for false positives, so a slightly lower precision is an acceptable trade-off. Accuracy is a poor choice here because medical datasets are usually imbalanced; a model that predicts negative for every image can score 98 percent accuracy while missing every sick patient. Precision would be the right answer in the opposite setup — for example, an automated action that cannot be reviewed — but not when a human reviewer already catches false positives. Mean squared error is a regression metric and does not apply to a binary classification output.
Why the other options are wrong
Accuracy is misleading on imbalanced datasets, which medical screening almost always is. A model that predicts "no disease" for every patient can post high accuracy while missing every sick person. The exam penalizes picking accuracy when the cost of false negatives and false positives is asymmetric.
Precision minimizes false positives, which matters when a positive prediction triggers an irreversible or expensive action with no human review. Here a radiologist already re-checks every flagged image, so false positives are cheap. Optimizing precision would cause the model to miss sick patients — the outcome the clinical team most wants to avoid.
Mean squared error is a regression metric used when the target is a continuous number, such as predicted house price or temperature. It is not defined for binary classification outputs and will never appear as the right answer on a screening or fraud-detection scenario.
How to choose a classification metric on AIF-C01
Every classification metric is a different view of the confusion matrix, which has four cells: true positives, false positives, true negatives, and false negatives. Accuracy is the share of all predictions that are correct. It is intuitive but breaks down the moment your classes are imbalanced. Precision is true positives divided by all predicted positives, so it answers the question, when the model says yes, how often is it right. Recall is true positives divided by all actual positives, so it answers, of all the real positives, how many did we catch. F1 is the harmonic mean of precision and recall and is a good default when you care about both and the classes are skewed. AUC-ROC measures how well the model ranks positives above negatives across every threshold and is a threshold-independent summary of separability. The exam decision rule is to ask which error is more painful. If a false positive is expensive, such as auto-blocking a paying customer or sending an aggressive treatment without review, prioritize precision. If a false negative is expensive, such as missing fraud, missing disease, or missing a safety defect, prioritize recall. If both matter and the data is imbalanced, use F1. For generative AI the story changes. Text generation is scored with BLEU or ROUGE against reference text, language models are scored with perplexity, and Amazon Bedrock model evaluation supports both automatic metrics and human evaluation jobs. Amazon SageMaker Clarify adds bias and explainability checks on top of these metrics so you can see not just how well the model performs overall, but also how performance differs across groups.
Ready to see how you'd score?
Take the free practice quiz and find out which AWS Certified AI Practitioner domains you need to focus on. No signup required.
Practice 5 similar questions
Same cert, same or adjacent domain. Use these after reviewing the explanation.