When to Fine-tune, Prompt, or Use RAG — A Decision Framework

Three months into building AlgoVesta’s trading signal extraction layer, we burned $8,000 on fine-tuning GPT-3.5 across 15,000 financial documents. The accuracy improved 3%. Then we switched to a different prompting approach and RAG architecture — same data, same models, different structure. Accuracy jumped to 89%. The problem wasn’t the technique. It was using the wrong technique for the problem.

Fine-tuning, prompt engineering, and retrieval-augmented generation (RAG) are not interchangeable tools competing for the same job. They solve different problems, operate at different costs, and require different maintenance burden. Most teams pick one because it sounds familiar or because they read a tutorial, not because it matches their actual constraints.

This framework cuts through that. It’s built on repeated production failures — mine and others — not on theoretical comparisons.

The Core Differences: What Each Actually Does

Start here because the distinctions matter.

Prompt engineering shapes the model’s behavior without changing its weights. You write instructions, structure examples, and design the input format. The model itself stays frozen. Cost is per-token. Latency is deterministic. Changes take seconds.

Fine-tuning updates the model’s weights by training it on your specific data. You create a dataset, run a training job (hours to days), and deploy a new model version. Cost includes training time upfront, then per-token inference. It’s permanent unless you retrain.

RAG doesn’t modify the model at all. It retrieves relevant context from an external database, injects it into the prompt, and lets the base model respond with grounded information. Cost depends on retrieval speed and the size of context injected. It scales with your knowledge base size, not your training data.

The distinction looks obvious when written out. In practice, teams confuse them constantly. A common error: someone trains a fine-tuned model to “improve accuracy” when the real problem is that the prompt wasn’t specific enough. Another: building a RAG system when fine-tuning would be faster and cheaper.

When Prompt Engineering Is Enough (And When It Stops Working)

Prompt engineering is your first move. Always.

It works when the base model already understands the concept and just needs better instructions. It costs nothing upfront. You get feedback in seconds. You can iterate 20 times in an hour. For classification, summarization, and structured extraction from text the model has seen during pretraining, prompt engineering often solves 70–85% of the problem.

Here’s a real before/after from extracting risk factors from SEC filings using Claude Sonnet 4:

## Bad prompt
Extract the main risk factors from this document.

## Output
- General market conditions could impact our business
- We face competition from other companies
- Regulations may change
- We depend on key personnel
(Vague, misses specificity, conflates severity levels)

## Improved prompt
From the attached 10-K filing, extract all risk factors that appear
in the "Item 1A. Risk Factors" section. For each risk:

1. Include the exact heading from the filing
2. Provide a 1-sentence core claim (what bad thing could happen)
3. Flag if the company quantifies likelihood or impact (if so, include it)
4. Mark as: MARKET / OPERATIONAL / REGULATORY / STRATEGIC / FINANCIAL
5. If the filing explicitly states this risk declined vs prior year,
   note that

Omit boilerplate risks that appear in every 10-K (e.g., "general
 economic conditions"). Focus on risks specific to this company's business.

## Output
- Cybersecurity Breaches and Data Losses (OPERATIONAL):
  Could compromise customer data and trigger regulatory action.
  Company estimated exposure at "up to $500M" if "material breach"
  occurred.
- Loss of Key Partnerships (STRATEGIC):
  Specific to this company. Two of top 5 revenue channels depend
  on contracts renewing annually.
(Specific, hierarchical, sortable)

The second prompt works because it:

• Specifies output structure (the model follows structured instructions better than vague requests)
• Adds negative examples (what NOT to include helps as much as what to include)
• Creates a classification system (MARKET/OPERATIONAL/etc. — this gives the model a decision tree)
• Quantifies precision (“1-sentence core claim” not “summarize the risk”)

Prompt engineering stops working when:

• The concept is outside the model’s training data. You can’t prompt your way around this. If you ask GPT-4o to extract domain-specific jargon from medical imaging reports and the model never saw that data during pretraining, it will guess.
• You need consistent behavior across 10,000+ similar queries. Prompt variance compounds. Some requests will hit edge cases and the prompt won’t handle them. You need systematic retraining of behavior.
• Your accuracy floor is ~85–90% but you need 96%+. Prompt engineering plateaus. Small gains after that require either fine-tuning or architectural changes (like RAG).
• The task requires facts not in the training data. If you’re extracting 2024 product pricing and your model was trained through April 2023, prompting won’t help. RAG will.

RAG: When Your Problem Is Missing or Outdated Context

RAG solves a specific, common problem: the model knows how to think but lacks the facts it needs.

You build a RAG system when:

• Your knowledge updates faster than model training cycles. If you retrain a fine-tuned model every week to add new data, RAG is cheaper. You update a vector database instead. Cost per query is higher (retrieval + embedding lookup) but you avoid retraining overhead.
• Your accuracy is failing because of hallucinations, not reasoning. The model knows how to extract information and structure it — it just invents details when it doesn’t know something. RAG grounds it in actual data.
• You need attribution. “Which document did this fact come from?” Fine-tuning can’t answer that. RAG can because it explicitly retrieves the source.
• Your task involves facts the model genuinely never saw. A fine-tuned model can’t learn something that wasn’t in its training data. A RAG system can retrieve it from a database.

Real example from AlgoVesta: we process 500+ research reports daily and extract buy/sell signals. The reports reference specific metrics and dates. Prices and date context shift constantly. We built RAG by:

1. Chunking each report into 300–500 token sections (semantic boundaries matter — splitting mid-sentence kills context)
2. Embedding each chunk with OpenAI’s text-embedding-3-small (cheap, fast, good enough for financial data)
3. Storing embeddings + original text in a vector database (Pinecone, in our case)
4. At query time: embed the user’s question, retrieve top-5 semantically similar chunks, inject them into the prompt, send to Claude Sonnet 4

The key metric: without RAG, the model hallucinated specific numbers ~15% of the time. With RAG, 1.2%. We didn’t fine-tune. We didn’t improve the prompt (much). We just gave the model the source material it needed.

RAG fails when:

• Your retrieval logic is broken. If your vector database returns irrelevant chunks, the model will work with bad input. You’ve just moved the problem upstream. You need to validate retrieval quality — actually look at top-5 results for a sample of queries.
• You’re injecting too much context. Claude Sonnet 4’s context window is 200K tokens. Sounds infinite until you’re injecting 15 chunks × 500 tokens each, plus a long prompt, plus the user’s request. Context bloat causes latency and cost spikes. More context doesn’t always improve accuracy — it can introduce noise.
• Your embedding model doesn’t match your domain. If you use a general-purpose embedding model on highly specialized data (legal contracts, medical records, trading signals), semantic similarity breaks down. You might need domain-specific embeddings or fine-tuned retrievers.
• Your knowledge base is small or static. If you have 50 documents that rarely change and your data is from the model’s training period, RAG adds complexity for minimal gain. Prompt engineering is faster.

Fine-tuning: When You Need Systematic Behavior Change

Fine-tuning is your last resort — not because it’s weak, but because it’s expensive and creates a deployment burden.

You fine-tune when:

• The base model struggles with your specific format or domain consistently. If prompt engineering gets you to 75% and you’ve exhausted prompt variations, fine-tuning can push you to 88–92% by reweighting the model toward your data distribution.
• You need latency guarantees that prompt engineering can’t provide. A fine-tuned model is smaller, faster, and more predictable than base models with long, complex prompts. If you’re serving 10K queries/sec and latency SLAs are tight, this matters.
• Cost per query is your constraint, not upfront cost. Fine-tuning is expensive upfront ($50–$500 depending on dataset size and model) but per-token inference is cheaper than base model inference. If you’re running high volume, the math flips.
• You need consistent behavior on edge cases your prompt can’t cover. If you’ve built a 2,000-token prompt trying to handle 12 different input variations and it still fails 8% of the time, a fine-tuned model trained on those variations will be more robust.

Example: a customer support team using GPT-4o to classify tickets. Prompt engineering got them to 82% accuracy across 50 ticket categories. They manually labeled 2,000 tickets, fine-tuned Mistral 7B (smaller model, faster inference), and hit 91% accuracy. Upfront: 4 hours of labeling + $60 fine-tuning cost. Monthly savings: $1,200 in API calls (fewer GPT-4o tokens needed). Payback: 3 weeks.

Fine-tuning fails when:

• You don’t have high-quality training data. If your labeled dataset is small (<500 examples) or noisy, fine-tuning will overfit or learn spurious patterns. Prompt engineering is more robust with limited data.
• Your problem is domain-specific and your model is too general. Fine-tuning GPT-4o on 1,000 molecular structure classification examples won’t teach it chemistry. A domain-specific model or specialized embedding approach would be better.
• Your data distribution shifts frequently. Fine-tuned models degrade when they encounter data unlike their training set. If you fine-tune today and your input patterns change next month, you need to retrain. RAG handles distribution shift better because you update the knowledge base, not the model.
• You need to deploy multiple variants. If you fine-tune one model for English and one for Spanish and one for a different domain, you now manage three separate models, three separate deployments, three separate monitoring pipelines. This operational overhead is real.

The Decision Matrix: Concrete Guidance for Your Problem

Use this table to shortcut the analysis. Find your constraint on the left, follow the row, and it points to the right tool.

A Real Decision Flow: Three Case Studies

Case 1: Legal Contract Risk Flagging

Startup needs to scan incoming contracts and flag unusual terms. They have 200 labeled contracts. Their base model (Claude Sonnet 4) catches 70% of risks but misses domain-specific language.

Decision: Start with prompt engineering (2 hours, free). Results: 78%. Then add RAG using their labeled contracts as a knowledge base (8 hours setup, $50 embedding costs). Results: 86%. Accuracy ceiling feels near. Fine-tuning would require 1,000+ labeled contracts (weeks of work). Decision: stop at RAG. Cost: ~$2/month in vector database. This works for 3–6 months until they have more labeled data.

Case 2: Product Review Classification (50K reviews/month)

SaaS company needs to classify 50K monthly reviews into 12 sentiment categories. GPT-4o API costs $400/month for the task. They label 3,000 examples and fine-tune Mistral 7B. Cost: $120 fine-tuning + $8/month inference (using Together AI for cheap inference). Results: 88% accuracy (vs. 82% with prompt engineering). Monthly savings: $370. Fine-tuning ROI: 4 weeks.

Decision: fine-tuning. But here’s the catch: they redeploy a new fine-tune every month as new reviews arrive. After 6 months they’ve run 6 fine-tuning cycles. At month 7, the newest version performs worse on old patterns (drift). They should have invested in RAG or a refreshed prompting strategy instead. Live and learn.

Case 3: Q&A Over Company Documentation (1,000+ pages)

HR department wants employees to ask questions about benefits, policies, and procedures. They have a 1,500-page handbook that updates quarterly. Base model (GPT-4o) hallucinates policy details 12% of the time.

Option A: Prompt engineering. Inject the entire handbook into the prompt. Results: 100KB+ token overhead per query. Latency: 4–5 seconds. Cost: $0.80 per query. Hallucinations: 5%. Untenable at scale.

Option B: RAG. Chunk the handbook (1,500 pages = ~1,000 chunks). Embed once. At query time: retrieve top-5 relevant chunks, inject them (~2KB tokens), send to model. Latency: 800ms. Cost: $0.12 per query. Hallucinations: 0.8%. Decision: obvious. Build RAG. Implementation: Pinecone + Claude Sonnet 4 = two weeks start-to-finish.

Hybrid Approaches: When One Technique Isn’t Enough

The real world rarely picks one. Teams that succeed combine them strategically.

Prompt engineering + RAG (common, effective)

You use RAG to ground facts, then use prompt engineering to guide reasoning. Example: Extract risk factors from a 10-K (RAG retrieves relevant sections) and then classify their severity using a structured prompt template (prompt engineering). Cost: retrieval overhead + inference. Latency: acceptable. Accuracy: high.

Fine-tuning + RAG (advanced, expensive)

You fine-tune a model to understand your domain, then use RAG to give it current facts. Example: fine-tune a medical model to classify patient symptoms, but use RAG to ground it in current drug interaction data that wasn’t in training. Cost: high (two systems to maintain). Benefit: domain understanding + factual accuracy. Use case: high-stakes domains where both matter.

Prompt engineering + prompt engineering (cascading) (underrated)

You use a cheap model for initial filtering, then a powerful model for refined output. Example: GPT-3.5 classifies whether an email is support-related (cheap, fast), then Claude Sonnet 4 drafts a response only if it is (saves money). Cost: two API calls but 70% cheaper overall. Latency: acceptable if first model is fast. Works well for routing and filtering tasks.

Benchmarking Your Approach: What to Measure

You can’t pick the right technique without baseline metrics. Before building anything, measure:

• Accuracy on a holdout test set (50–100 examples you don’t train on). Accuracy is context-dependent — 80% might be great for some tasks, terrible for others. Know your target.
• Cost per query (tokens × model price). Include retrieval costs if using RAG. Include fine-tuning amortized over expected query volume if fine-tuning.
• Latency percentiles, not averages (p50, p95, p99). Average latency hides tail slowness. If 1% of queries take 30 seconds, it breaks user experience even if average is 2 seconds.
• Failure modes (not just overall accuracy). Classify your errors: hallucinations, reasoning mistakes, format errors, ambiguous input. Different techniques fail differently. Hallucinations point toward RAG. Format errors point toward better prompting or fine-tuning.
• Variance across input types (not one monolithic accuracy number). If your model works great on short inputs but fails on long ones, you have a structural problem, not a technique problem.

Example from AlgoVesta: we measured trading signal extraction accuracy not as a single number but as:

• Accuracy on recent reports (2024): 91%
• Accuracy on archived reports (2020–2022): 73% (knowledge gap)
• Accuracy when price data is referenced: 88%
• Accuracy on edge-case tickers: 62% (rare stocks, hallucinations spike)

This breakdown revealed that RAG solved the 2020–2022 problem and the price reference problem (they’re in the knowledge base), but didn’t solve the edge-case ticker problem (model needs fine-tuning or better ticker disambiguation in the prompt). One number wouldn’t have told us that.

What to Do Starting Monday

If you’re facing this decision now:

Step 1: Pick a small sample (10–50 examples) from your use case and measure baseline accuracy with the cheapest option first: GPT-3.5 Turbo with a simple prompt. This takes 30 minutes and gives you a floor. Might be good enough. Often isn’t.

Step 2: Write a more specific prompt (reference the section earlier for prompt template rules). Measure again.** Usually 5–15% improvement with zero cost. Stop if this achieves your target accuracy. Don’t over-engineer.