CoursesFree Quizzes
NVIDIAEvaluation

Reproducible Evaluation & Determinism — NCP-GENL Practice Question

A representative NVIDIA-Certified Professional Generative AI LLMs (NCP-GENL) exam question on Reproducible Evaluation & Determinism. Work through it below, then read why each option is right or wrong.

This question is part of the free NVIDIA-Certified Professional Generative AI LLMs practice test.

Short answer

The correct answer is D. Use PyTorch with a fixed random seed, disable GPU non-determinism via CUDA environment variables, and store the exact model checkpoint, input data, and evaluation code version in a version control system.

Reproducibility requires controlling all sources of non-determinism: random initialization, GPU floating-point variance, and code changes. PyTorch with fixed seeds, disabled GPU non-determinism (via CUBLAS_WORKSPACE_CONFIG and torch.use_deterministic_algorithms), plus version control of model, data, and code is the standard reproducibility approach and directly addresses the regulatory audit requirement. NIM caching doesn't guarantee reproducibility if the model or data changes. Single-GPU FP32 is unnecessarily restrictive and doesn't address code versioning. CPU-only computation is impractical for LLM evaluation and doesn't address reproducibility of the model itself. The seeded, versioned PyTorch setup is the principled, scalable reproducibility pattern.

The Question

An insurance claims team is using a generative AI system to summarize claim narratives and identify missing information. Reproducible evaluation is critical: the team must be able to re-run the exact same evaluation on the same data and get identical results every time, to support regulatory audits and model improvement cycles. The architecture selection must ensure deterministic behavior across re-runs. Which approach best guarantees reproducibility?

ARun evaluation on a single GPU using TensorRT-LLM with FP32 precision; avoid multi-GPU evaluation to eliminate NCCL communication variability
BImplement a custom evaluation harness that freezes the model weights in memory, disables all dropout layers, and uses only CPU-based metric computation to avoid GPU floating-point variance
CDeploy the model using NIM (NVIDIA Inference Microservices) with caching enabled; cache all intermediate embeddings and attention outputs so that identical inputs always produce identical outputs
DUse PyTorch with a fixed random seed, disable GPU non-determinism via CUDA environment variables, and store the exact model checkpoint, input data, and evaluation code version in a version control systemCorrect

Why D is correct

Reproducibility requires controlling all sources of non-determinism: random initialization, GPU floating-point variance, and code changes. PyTorch with fixed seeds, disabled GPU non-determinism (via CUBLAS_WORKSPACE_CONFIG and torch.use_deterministic_algorithms), plus version control of model, data, and code is the standard reproducibility approach and directly addresses the regulatory audit requirement. NIM caching doesn't guarantee reproducibility if the model or data changes. Single-GPU FP32 is unnecessarily restrictive and doesn't address code versioning. CPU-only computation is impractical for LLM evaluation and doesn't address reproducibility of the model itself. The seeded, versioned PyTorch setup is the principled, scalable reproducibility pattern.

Why the other options are wrong

Option A: Run evaluation on a single GPU using TensorRT-LLM with FP32 precision; avoid multi-GPU evaluation to eliminate NCCL communication variability

Option A does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (D) is the only one that fully meets every constraint stated in the question.

Option B: Implement a custom evaluation harness that freezes the model weights in memory, disables all dropout layers, and uses only CPU-based metric computation to avoid GPU floating-point variance

Option B does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (D) is the only one that fully meets every constraint stated in the question.

Option C: Deploy the model using NIM (NVIDIA Inference Microservices) with caching enabled; cache all intermediate embeddings and attention outputs so that identical inputs always produce identical outputs

Option C does not satisfy the requirement in the scenario. Review the explanation above: the correct choice (D) is the only one that fully meets every constraint stated in the question.

Key idea: Reproducible Evaluation & Determinism

Reproducibility requires controlling all sources of non-determinism: random initialization, GPU floating-point variance, and code changes. PyTorch with fixed seeds, disabled GPU non-determinism (via CUBLAS_WORKSPACE_CONFIG and torch.use_deterministic_algorithms), plus version control of model, data, and code is the standard reproducibility approach and directly addresses the regulatory audit requirement. NIM caching doesn't guarantee reproducibility if the model or data changes. Single-GPU FP32 is unnecessarily restrictive and doesn't address code versioning. CPU-only computation is impractical for LLM evaluation and doesn't address reproducibility of the model itself. The seeded, versioned PyTorch setup is the principled, scalable reproducibility pattern. On the NCP-GENL exam, questions in the "Evaluation" domain test whether you can map a scenario's constraints to the right choice. Read the requirement carefully, eliminate options that violate any single constraint, and pick the one that satisfies all of them with the least operational overhead.

Ready to see how you'd score?

Take the free practice quiz and find out which NVIDIA-Certified Professional Generative AI LLMs domains you need to focus on. No signup required.

Take the free quizSee all 222+ practice questions

Practice 5 similar questions

Same cert, same or adjacent domain. Use these after reviewing the explanation.

1Multi Head Attention & Attention Specialization — NCP-GENL Practice QuestionLLM Architecture2Nemo Guardrails & Jailbreak Detection — NCP-GENL Practice QuestionSafety, Ethics, and Compliance3Chain-of-Thought & Output Control — NCP-GENL Practice QuestionPrompt Engineering4Nemo Curator & Semantic Deduplication — NCP-GENL Practice QuestionData Preparation5Quantization & Memory Optimization — NCP-GENL Practice QuestionModel Optimization

Related NVIDIA-Certified Professional Generative AI LLMs Practice Questions

Multi Head Attention & Attention Specialization — NCP-GENL Practice Question→Nemo Guardrails & Jailbreak Detection — NCP-GENL Practice Question→Chain-of-Thought & Output Control — NCP-GENL Practice Question→

Official sources used

Verify program and exam details at the source before scheduling.

docs.nvidia.comLLM: Index.htmlnvidia.comGenerative AI LLM Professional