"The model keeps making things up. Should we fine-tune it?"

Quick Answer

RAG (Retrieval Augmented Generation) fetches relevant documents at query time to augment the LLM prompt — best for knowledge bases that change frequently. Fine-tuning trains the model on new data to change its behavior — best for style, format, or specialized vocabulary. Prompt engineering costs nothing but has limits. RAG is the recommended default for enterprise knowledge applications.

RAGRetrieve documents via Azure AI Search → inject into system prompt → model answers using retrieved context. No model training needed.
Fine-tuningTrain base model on custom dataset to change its response style or add domain vocabulary. Expensive, needs labeled data.
When to fine-tuneWhen you need consistent output format, specific tone, or technical vocabulary that prompt engineering cannot reliably enforce
Certification relevanceAI-200 Domain: Design and implement a solution to ground and improve a generative AI solution

Priya's manager asked her this after a demo of their internal legal document assistant. The Azure OpenAI-powered chatbot had answered a contract question confidently — but completely incorrectly. It had fabricated a clause that didn't exist in their contracts.

"This is a hallucination problem," Priya said. "Fine-tuning won't fix it. We need RAG."

Her manager looked puzzled. "What's the difference?"

This is exactly the question AI-200 asks — in scenario form. Let's understand all three approaches deeply enough to answer any variation the exam throws at you.

🔤 Prompt Engineering — Shaping Behaviour Without Training

Prompt engineering is the art and science of crafting the input to an LLM to get the output you want — without touching the model's weights or providing external documents.

It works at three levels in Azure OpenAI:

  • System prompt: Sets the model's persona, tone, and constraints — "You are a legal assistant. Always respond in formal British English. Never speculate about facts not provided to you."
  • Few-shot examples: Providing example input-output pairs in the prompt to teach the model the expected format or style.
  • Parameter tuning: Setting temperature (creativity vs. determinism), top-p, max tokens, and stop sequences to shape the response characteristics.
Scenario
Prompt Engineering Solution
Responses too casual for a banking app
System prompt: "Always respond in formal professional language. Do not use contractions."
Outputs need to be valid JSON
System prompt + few-shot examples + low temperature forces consistent structured output
Model too creative, inconsistent answers
Set temperature to 0 — forces the model toward its most deterministic response
💡 What Prompt Engineering Cannot Do It cannot inject new facts into the model. If the model was not trained on your company's internal documents, no system prompt will give it that knowledge. It will hallucinate — as Priya discovered.

📚 RAG — Retrieval-Augmented Generation

RAG solves the hallucination problem by supplying the facts the model needs at query time. Instead of relying on what the LLM was trained on, you retrieve the most relevant documents from your own knowledge base and inject them into the prompt.

In Azure, a RAG architecture typically looks like this:

1
User asks a question
"What is the termination notice period in the Master Services Agreement?"
↓
2
Retrieve relevant chunks from Azure AI Search
Hybrid search (keyword + vector) retrieves the 3–5 most semantically relevant document chunks from your indexed contracts.
↓
3
Construct a grounded prompt
The retrieved chunks are injected into the Azure OpenAI prompt: "Answer the user's question using ONLY the following contract excerpts. Do not speculate beyond these documents."
↓
4
Grounded response with citations
The LLM answers based on your actual contract text — not its training data. It can cite the source chunk. Hallucination is eliminated because the answer is constrained to provided context.
✅ When RAG Is the Correct AI-200 Answer Whenever the scenario involves: company-specific documents, frequently changing content (policies, products, prices), compliance requirements for source citation, or reducing hallucinations on factual questions — the answer is RAG, not fine-tuning.

🔧 Fine-Tuning — Teaching the Model New Behaviour

Fine-tuning trains the LLM on your custom dataset, permanently adjusting its weights to produce different outputs. It is powerful — but it is also the most expensive, the slowest to iterate on, and often the wrong choice for problems that RAG or prompt engineering can solve.

Fine-Tuning IS Right
Fine-Tuning is NOT Right
Teaching a highly specific response style that prompt engineering cannot consistently achieve
The model doesn't know your internal data (use RAG instead)
Domain-specific classification: medical coding, legal clause extraction, semiconductor failure analysis
The data changes frequently — retraining monthly is expensive and slow
Reducing token usage by teaching concise formats without long few-shot examples each call
You just want the model to answer questions about your documents (RAG is faster and cheaper)
"Fine-tuning teaches the model HOW to respond. RAG gives the model WHAT to respond about. Mixing up these two is the most common mistake in AI-200 scenario questions."

📊 The Decision Framework — All Three Side by Side

Factor
Prompt Engineering
RAG
Fine-Tuning
Primary purpose
Shape tone, format, behaviour
Ground answers in your documents
Teach new response patterns
Fixes hallucinations?
❌ Partially
✅ Yes — primary strength
❌ No
Works with changing data?
✅ Yes
✅ Yes — re-index to update
❌ Costly to retrain
Cost
💰 Lowest
💰💰 Medium
💰💰💰 Highest
Implementation speed
⚡ Hours
⚡⚡ Days
⚡⚡⚡ Weeks
Exam preference
Format / style problems
Most enterprise scenarios
Domain-specific style adaptation

🧩 Real Scenario Mapping — What Would You Choose?

Apply the framework to these realistic AI-200 scenarios:

  • Scenario A: A bank wants its AI assistant to answer questions about its 2,000-page policy manual. The manual is updated quarterly.
    → RAG. Dynamic documents, factual Q&A, high hallucination risk.
  • Scenario B: A law firm wants an AI that always responds in a specific formal legal writing style. The model's default tone is too casual despite system prompts.
    → Fine-tuning. Style adaptation that prompt engineering cannot reliably achieve.
  • Scenario C: A customer service bot must always output a valid JSON object with fields: intent, sentiment, and suggested_response.
    → Prompt engineering. Structured output formatting with few-shot examples and low temperature.
  • Scenario D: A healthcare company needs a model that classifies clinical notes into ICD-10 codes. The classification pattern is highly specialised and not achievable through prompting alone.
    → Fine-tuning. Domain-specific classification with stable categories.
📚 Go Deeper — Microsoft Learn Resources
Coming Next AI-200 Scenario-Based Practice — Real Azure AI Architecture Questions with Full Explanations. Put the decision framework into action with exam-style scenarios covering Event Hubs, Private Endpoints, Managed Identity, and multi-service AI pipelines.