Fine Tuning & Continued Pre Training — AWS GenAI Dev Practice Question
A representative AWS Generative AI Developer (AWS GenAI Dev) exam question on Fine Tuning & Continued Pre Training. Work through it below, then read why each option is right or wrong.
Short answer
The correct answer is A. Perform continued pre-training on the 50,000 unlabeled legal documents first, then fine-tune the resulting model with the 5,000 labeled examples for clause extraction.
This is the correct two-stage approach for domain adaptation. Continued pre-training on unlabeled data teaches the model domain-specific language patterns, terminology, and document structures. Fine-tuning on labeled data then teaches the model the specific task (clause extraction). This sequential approach leverages both data types optimally and is a well-established pattern in foundation model customization on Bedrock.
The Question
A legal technology company wants to improve a foundation model's understanding of specialized legal terminology and document structures that are not well-represented in the base model's training data. They have 50,000 unlabeled legal documents. After improving the model's legal domain understanding, they also need to train it to extract specific clause types from contracts using 5,000 labeled examples. The company wants to minimize costs while achieving the best domain-specific performance. Which approach should the developer take?
Why A is correct
This is the correct two-stage approach for domain adaptation. Continued pre-training on unlabeled data teaches the model domain-specific language patterns, terminology, and document structures. Fine-tuning on labeled data then teaches the model the specific task (clause extraction). This sequential approach leverages both data types optimally and is a well-established pattern in foundation model customization on Bedrock.
Why the other options are wrong
Fine-tuning only on labeled examples without domain adaptation means the model lacks understanding of specialized legal terminology. Including documents in the system prompt is limited by context window size and significantly increases per-request inference costs, especially with 50,000 documents.
Combining labeled and unlabeled data for continued pre-training wastes the labeled data's value. Continued pre-training is unsupervised and doesn't use labels, so the 5,000 labeled examples would lose their annotation information. The fine-tuning step is necessary to learn the clause extraction task.
While LoRA reduces fine-tuning costs, using RAG with 50,000 legal documents for domain understanding is suboptimal compared to continued pre-training. RAG retrieves relevant chunks at query time but doesn't fundamentally improve the model's understanding of legal language patterns. For deep domain adaptation, continued pre-training is more effective.
Key idea: Fine Tuning & Continued Pre Training
Why A is correct: This is the correct two-stage approach for domain adaptation. Continued pre-training on unlabeled data teaches the model domain-specific language patterns, terminology, and document structures. Fine-tuning on labeled data then teaches the model the specific task (clause extraction). This sequential approach leverages both data types optimally and is a well-established pattern in foundation model customization on Bedrock. Why B is wrong: Fine-tuning only on labeled examples without domain adaptation means the model lacks understanding of specialized legal terminology. Including documents in the system prompt is limited by context window size and significantly increases per-request inference costs, especially with 50,000 documents. Why C is wrong: Combining labeled and unlabeled data for continued pre-training wastes the labeled data's value. Continued pre-training is unsupervised and doesn't use labels, so the 5,000 labeled examples would lose their annotation information. The fine-tuning step is necessary to learn the clause extraction task. Why D is wrong: While LoRA reduces fine-tuning costs, using RAG with 50,000 legal documents for domain understanding is suboptimal compared to continued pre-training. RAG retrieves relevant chunks at query time but doesn't fundamentally improve the model's understanding of legal language patterns. For deep domain adaptation, continued pre-training is more effective. On the AWS GenAI Dev exam, questions in the "Foundation Model Integration, Data Management, and Compliance" domain test whether you can map a scenario's constraints to the right choice. Read the requirement carefully, eliminate options that violate any single constraint, and pick the one that satisfies all of them with the least operational overhead.
Ready to see how you'd score?
Take the free practice quiz and find out which AWS Generative AI Developer - Professional domains you need to focus on. No signup required.
Practice 5 similar questions
Same cert, same or adjacent domain. Use these after reviewing the explanation.