Evaluation Metrics & BLEU — NCA-GENL Practice Question
A representative NVIDIA Generative AI LLMs Associate (NCA-GENL) exam question on Evaluation Metrics & BLEU. Work through it below, then read why each option is right or wrong.
Short answer
The correct answer is A. BLEU relies on exact n-gram overlap with references and penalizes valid paraphrases.
BLEU measures precision of n-gram overlap between generated and reference text. When a translation is semantically correct but uses different vocabulary or syntax, BLEU penalizes it because there is little exact n-gram match with the reference. This is a well-known limitation of BLEU. Option B is not the core issue here. Option C is incorrect because BLEU is typically computed at the corpus level. Option D is wrong because BLEU actually includes a brevity penalty to address length differences.
The Question
An engineer is evaluating an LLM's machine translation output using the BLEU metric. The model produces fluent translations that convey the correct meaning but uses different word choices and sentence structures compared to the reference translations. What limitation of BLEU does this scenario highlight?
Why A is correct
BLEU measures precision of n-gram overlap between generated and reference text. When a translation is semantically correct but uses different vocabulary or syntax, BLEU penalizes it because there is little exact n-gram match with the reference. This is a well-known limitation of BLEU. Option B is not the core issue here. Option C is incorrect because BLEU is typically computed at the corpus level. Option D is wrong because BLEU actually includes a brevity penalty to address length differences.
Why the other options are wrong
Option B does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (A) is the only one that fully meets every constraint stated in the question.
Option C does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (A) is the only one that fully meets every constraint stated in the question.
Option D does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (A) is the only one that fully meets every constraint stated in the question.
Key idea: Evaluation Metrics & BLEU
BLEU measures precision of n-gram overlap between generated and reference text. When a translation is semantically correct but uses different vocabulary or syntax, BLEU penalizes it because there is little exact n-gram match with the reference. This is a well-known limitation of BLEU. Option B is not the core issue here. Option C is incorrect because BLEU is typically computed at the corpus level. Option D is wrong because BLEU actually includes a brevity penalty to address length differences. On the NCA-GENL exam, questions in the "Experimentation and Evaluation" domain test whether you can map a scenario's constraints to the right choice. Read the requirement carefully, eliminate options that violate any single constraint, and pick the one that satisfies all of them with the least operational overhead.
Ready to see how you'd score?
Take the free practice quiz and find out which NVIDIA Generative AI LLMs Associate domains you need to focus on. No signup required.
Practice 5 similar questions
Same cert, same or adjacent domain. Use these after reviewing the explanation.