Zero-Shot vs Few-Shot vs Chain-of-Thought: Pick the Right Technique

Last month, Claude Sonnet 4 failed to extract structured data from a customer invoice on the first try. No examples. No reasoning steps. Just a straightforward instruction. Then I added one invoice example to the prompt. It worked. Then I asked it to think through the extraction logic step-by-step before answering. It worked better.

Three different prompting approaches. Three different outcomes. The difference wasn’t in the model — it was in the cognitive scaffolding I built into the prompt itself.

This is where most teams fail. They treat prompting like a binary choice: “Does this work or not?” They don’t ask the harder question: “Which of these three methods will actually solve this problem with the fewest tokens and the highest accuracy?” That’s the real decision.

This article breaks down zero-shot, few-shot, and chain-of-thought prompting — not as abstract concepts, but as concrete techniques you’ll choose between in production systems. You’ll see exactly when each works, when each fails, and how to combine them into a coherent strategy.

The Three Techniques at a Glance

Before we dig into specifics, here’s the landscape:

Zero-shot prompting means you give the model a task with no examples. Just instructions and context. Fast. Token-cheap. Works for straightforward tasks.

Few-shot prompting means you give the model 2–5 examples of the task before asking it to solve the problem. More context. Higher accuracy. Costs more tokens.

Chain-of-thought prompting means you ask the model to explain its reasoning step-by-step before arriving at an answer. Slower. More expensive. But it reduces hallucination and improves reasoning on complex problems.

Here’s a comparison table with real-world performance data from my own testing across invoice extraction, customer classification, and bug categorization tasks:

These numbers matter. They shape the real economics of your system — token cost, latency, accuracy trade-offs. But they’re also context-dependent. A 76% zero-shot accuracy might be fine for customer sentiment tagging. It’s unacceptable for loan approval decisions.

Zero-Shot: When You Can Afford to Keep It Simple

Zero-shot is the baseline. No examples. No reasoning request. Just the instruction and the input.

Here’s a concrete example. You’re building a customer support classifier that routes incoming tickets to the right team:

# Bad zero-shot prompt (too vague)

Classify this support ticket:

"My shipment hasn't arrived in 3 weeks. I'm frustrated."

Category: 

The model might output “Shipping” or “Complaints” or “Urgent”. You don’t know. The instruction is ambiguous about what categories exist or how to distinguish between them.

Here’s the improved version:

# Improved zero-shot prompt (explicit instruction)

You are a support ticket router. Classify the following ticket into ONE of these categories:
- Billing: payment issues, invoice disputes, refund requests
- Shipping: delivery delays, tracking issues, address problems
- Product Quality: defects, damage, returns
- Account: login issues, password resets, account access
- Other: any other issue

Ticket: "My shipment hasn't arrived in 3 weeks. I'm frustrated."

Reasoning: 
Category: 

Now you get “Shipping” consistently. The difference: explicit categories + clear criteria for each + a slot for reasoning (which the model uses even though you didn’t ask for chain-of-thought reasoning).

Zero-shot works best when:

• The task is familiar to the model’s training data. Sentiment analysis, basic classification, summarization — these are common enough that the model has learned implicit patterns. It doesn’t need examples to remember how to do them.
• Token budget is tight. You’re processing thousands of requests per day and every token adds up. A zero-shot approach saves 500–800 tokens per request. That’s real money.
• Accuracy requirements are moderate. You need 70–80% accuracy, and you have a fallback workflow for edge cases (human review, retry with a different model, etc.)
• The output format is simple. A single category. A yes/no decision. A one-sentence summary. If you need structured JSON or multi-step output, zero-shot fails more often.

Zero-shot fails spectacularly when:

• Domain-specific terminology matters. If you’re asking the model to categorize medical conditions or financial instruments, and the categories are narrow or overlapping, zero-shot guesses. GPT-4o without examples gets domain-specific classification right maybe 65% of the time.
• The output format is custom. You need the model to output a specific JSON structure, or a markdown table, or a numbered list with bold headings — the model will get close but often miss details.
• Consistency is critical. If you’re using the model output as training data for another system, or as reference material for end users, zero-shot’s variability becomes a liability.

Few-Shot: Trading Tokens for Accuracy

Few-shot prompting means you show the model 2–5 examples of the task, then ask it to perform the same task on new input.

The example accuracy boost is real. I’ve measured it repeatedly, and the pattern holds: adding 2–4 well-chosen examples typically lifts accuracy by 8–15 percentage points. Diminishing returns kick in after 5 examples. Adding 10 examples doesn’t double your accuracy — it adds maybe 2–3 more percentage points while burning 800 extra tokens.

Here’s the invoice extraction example I mentioned earlier. Zero-shot version first:

# Zero-shot invoice extraction (fails ~35% of the time)

Extract the following fields from the invoice:
- Invoice Number
- Invoice Date
- Total Amount
- Due Date
- Customer Name

Return as JSON.

Invoice:
[invoice text]

JSON: 

The model extracts something. But it misses the due date 30% of the time, or puts the amount in the wrong field, or formats the date as “12/25/2024” when your system expects “2024-12-25”.

Now the few-shot version with 3 examples:

# Few-shot invoice extraction (succeeds ~82% of the time)

Extract the following fields from invoices. Return as valid JSON only, no other text.

Example 1:
Invoice text: "Invoice #INV-2024-001 issued on January 15, 2024. Bill to Acme Corp. Total: $5,500.00. Payment due by February 15, 2024."
JSON: {"invoice_number": "INV-2024-001", "invoice_date": "2024-01-15", "customer_name": "Acme Corp", "total_amount": "5500.00", "due_date": "2024-02-15"}

Example 2:
Invoice text: "Reference: SVC-2024-087. Service invoice dated 01/20/2024. For: TechStart Inc. Amount due: $3,250.50. Due on: 02/20/2024. This invoice covers Q1 consulting services."
JSON: {"invoice_number": "SVC-2024-087", "invoice_date": "2024-01-20", "customer_name": "TechStart Inc", "total_amount": "3250.50", "due_date": "2024-02-20"}

Example 3:
Invoice text: "Invoice # 12345 dated Dec 1, 2023. Customer: Global Solutions Ltd. Total invoice amount: $7,890.25. Payment required by: January 1, 2024."
JSON: {"invoice_number": "12345", "invoice_date": "2023-12-01", "customer_name": "Global Solutions Ltd", "total_amount": "7890.25", "due_date": "2024-01-01"}

Now extract from this invoice:
Invoice text: "[new invoice text]"
JSON: 

The examples show the model exactly what you want: the date format (YYYY-MM-DD), the amount format (decimal only, no currency symbol), the JSON structure, and how to handle variations in the source text (“Reference:” vs “Invoice #”, “due on:” vs “Payment required by:”).

The accuracy jump is substantial. With these examples, the same invoice extraction task succeeds ~82% of the time instead of ~65%.

Few-shot works best when:

• Output format is custom or specific. JSON structure, CSV format, markdown table — examples show the model the exact pattern you need.
• Domain terminology is narrow or specialized. If you’re classifying legal document types, or categorizing medical conditions, or segmenting financial products, 3–4 examples of each category teach the model your taxonomy faster than instructions alone.
• Edge cases exist and matter. Maybe most invoices have a due date, but some don’t. Maybe most customer names are obvious, but some are embedded in a long legal name. Examples surface these patterns.
• Token budget allows it. If you’re processing thousands of requests and token cost is secondary to accuracy, few-shot is reasonable. If you’re optimizing for sub-100ms latency, few-shot adds 1–2 seconds of prompt processing.

Few-shot fails or becomes uneconomical when:

• You have more than 10 categories or highly variable outputs. You’d need 20–50 examples to cover all cases. At that point, you’re spending 3,000+ tokens just on examples. The accuracy gain plateaus.
• The task is truly novel. If the model has never seen this type of problem in training data, examples help — but not as much. Chain-of-thought becomes more valuable.
• Your examples are bad or unrepresentative. If you cherry-pick examples that are easier than real data, the model learns the wrong pattern. This is subtle and deadly. I’ve seen few-shot prompts that work great on internal test data but fail 40% of the time on production data because the test examples weren’t representative.

Chain-of-Thought: Making Reasoning Visible

Chain-of-thought prompting asks the model to explain its reasoning step-by-step before answering. It sounds obvious — “Show your work” — but the results are profound.

The mechanism: when you force the model to articulate intermediate steps, it catches its own errors. If it’s about to make a wrong inference, it catches it while “thinking.” If it’s hallucinating a fact, the reasoning often exposes the fabrication.

Research from Wei et al. (2022) showed that chain-of-thought prompting improved performance on math word problems from 17% (GPT-3) to 78% (GPT-3 + CoT). That’s not a marginal improvement. That’s transformative within the scope of that task.

A practical example. You’re building a customer value calculator. Given a customer’s purchase history, usage patterns, and churn risk, you need to assign them to a segment: “High-Value Stable”, “High-Value At Risk”, “Medium-Value Growth”, or “Low-Value Churn Risk”.

Zero-shot:

# Zero-shot segmentation (inconsistent, ~68% confidence)

You are a customer success analyst. Segment this customer based on their profile:

Customer Profile:
- Annual Spend: $45,000
- Usage (% of available): 62%
- Support Tickets (12 months): 8
- NPS Score: 32
- Product Adoption: 4 of 8 modules
- Churn Risk Score: 0.71

Segment (choose one): High-Value Stable, High-Value At Risk, Medium-Value Growth, Low-Value Churn Risk
Segment: 

The model might output “High-Value At Risk” or “Medium-Value Growth”. Both are defensible. You don’t know the reasoning, so you can’t debug failures.

Chain-of-thought version:

# Chain-of-thought segmentation (consistent, ~87% confidence)

You are a customer success analyst. Segment this customer based on their profile. Explain your reasoning step-by-step, then provide your final segment.

Customer Profile:
- Annual Spend: $45,000
- Usage (% of available): 62%
- Support Tickets (12 months): 8
- NPS Score: 32
- Product Adoption: 4 of 8 modules
- Churn Risk Score: 0.71

Analyze the customer's value and risk:
1. Revenue Potential: Is their annual spend high, medium, or low relative to your customer base?
2. Engagement Level: Are they actively using the product (usage %) or passive?
3. Satisfaction: What does NPS and support ticket volume tell you about satisfaction?
4. Growth Trajectory: Are they adopting more modules, or stalled?
5. Risk Level: What does the churn risk score suggest?

Then provide your final segment: High-Value Stable, High-Value At Risk, Medium-Value Growth, Low-Value Churn Risk

Reasoning:
1. Revenue Potential: $45k is medium-to-high for most B2B SaaS. This customer is a valuable account.
2. Engagement Level: 62% usage suggests active engagement but with untapped potential. They're not maximizing the platform.
3. Satisfaction: NPS of 32 is neutral-to-negative (0–30 is detractor, 31–70 is passive). 8 support tickets in 12 months is moderate — not overly high, not perfectly smooth.
4. Growth Trajectory: Using 4 of 8 modules shows they've adopted core features but haven't expanded to adjacent modules. There's growth potential.
5. Risk Level: Churn risk score of 0.71 (on 0–1 scale) is high. This customer is at risk of leaving.

Final Segment: 

Now the model outputs “High-Value At Risk” with visible reasoning. You can audit the logic. If the output is wrong, you know where in the reasoning chain the error occurred — and you can adjust the prompt accordingly.

More importantly, the accuracy improves. Chain-of-thought prompting on this task (I’ve tested it multiple times) lifts accuracy from ~68% to ~87%. The model catches edge cases because it’s forced to articulate why.

Chain-of-thought works best when:

• The task involves multi-step reasoning. If you need the model to weigh multiple factors, make inferences, or connect dots across different data points, CoT shines.
• The stakes justify the extra latency and tokens. Chain-of-thought adds 3–5 seconds of latency (due to longer output tokens) and burns 800–1,200 extra tokens. That’s acceptable for high-stakes decisions (loan approvals, hiring recommendations, medical triage) but excessive for high-volume, low-stakes tasks (tagging support tickets, categorizing emails).
• Explainability is a requirement. If you need to show end users or regulators why a decision was made, CoT output is invaluable. You can’t explain “the model said so.” You can explain step-by-step reasoning.
• Hallucination is a known problem with your task. If the model tends to invent facts or miss important context, forcing it to reason step-by-step exposes those gaps. It’s not a perfect fix, but it helps.

Chain-of-thought fails or backfires when:

• The task is simple or straightforward. If you’re asking “Is this email about billing?” (yes/no), forcing the model to produce 5 paragraphs of reasoning is wasteful. It adds latency and tokens without meaningful accuracy gains.
• Latency is critical. Real-time systems — chatbots, API-driven workflows, live autocomplete — can’t wait 4–5 seconds for CoT reasoning. The user experience suffers.
• The model doesn’t actually reason. If the model is confabulating (making up reasoning that sounds plausible but isn’t grounded in the input), CoT just produces longer confabulation. You have to pair CoT with other techniques — like retrieval-augmented generation (RAG) — to ground the reasoning.

Combining Techniques: Few-Shot + Chain-of-Thought

The highest-accuracy systems combine few-shot with chain-of-thought. You show examples, then ask the model to reason step-by-step before answering.

This is expensive in tokens but worth it for high-stakes decisions.

Here’s a loan approval decision example (simplified for clarity):

# Few-shot + Chain-of-thought loan approval

You are a loan underwriter. Decide whether to approve or deny a loan application. Show your reasoning step-by-step, then provide your decision and confidence.

Example 1:
Applicant: Software engineer, employed 5 years, $120k salary, $50k savings, credit score 750, debt-to-income 22%, no recent delinquencies.
Loan Amount: $200,000 (mortgage for primary residence).
Reasoning: Income is stable and high. Down payment ($50k) is 20% of home value. Credit score is good. Debt-to-income is healthy. No red flags.
Decision: APPROVE. Confidence: 92%

Example 2:
Applicant: Freelancer, variable income ($60k–$90k annually), $5k savings, credit score 620, debt-to-income 45%, three late payments in past 24 months.
Loan Amount: $150,000 (personal loan).
Reasoning: Income is unstable. Savings are minimal (only 3% of loan amount). Credit score is below 650 threshold. High debt-to-income. Recent payment history is concerning. Multiple risk factors present.
Decision: DENY. Confidence: 88%

Now evaluate this applicant:
Applicant: Marketing director, employed 3 years, $85k salary, $15k savings, credit score 710, debt-to-income 28%, one late payment 18 months ago (paid).
Loan Amount: $120,000 (home improvement loan, secured by primary residence).

Reasoning:
1. Income Stability: Employed 3 years in stable role. Salary is $85k. This is reasonable income for the loan amount.
2. Down Payment / Savings: $15k in savings. For a $120k loan, that's 12.5% — modest but acceptable for a secured loan.
3. Credit Score: 710 is good. Not excellent, but above 700 threshold.
4. Debt-to-Income: 28% is within acceptable range (typically up to 35–43%).
5. Payment History: One late payment 18 months ago, but it was paid. Recent history is clean (18 months without issues). This suggests recovery from a temporary problem.
6. Loan Type: Secured by primary residence, which reduces risk.

Decision: APPROVE. Confidence: 78%

Explanation: This applicant has reasonable income, acceptable credit, manageable debt, and a positive recent payment history. The secured nature of the loan provides additional protection. Risk is moderate but acceptable.

The combination works because:

• Few-shot teaches the model your criteria. What you consider “good” credit, “stable” income, “acceptable” debt-to-income. The examples embed your decision logic.
• Chain-of-thought makes the logic explicit. You can see exactly which factors the model weighted and how. If a decision is wrong, you can trace the error.
• Confidence scores are more reliable. When the model is forced to reason, its confidence estimates correlate better with actual accuracy.

The cost is real: 1,400–2,200 tokens per request, $7.50–$12.00 per 1M requests. That’s 10–15x the cost of zero-shot. But if a single wrong decision costs $10,000+ (a bad loan approval, a missed customer opportunity, a safety incident), the math is trivial.

The Decision Framework: Which Technique to Use

Here’s how to choose in practice:

Start with accuracy requirements. What’s the cost of a wrong answer? If a wrong answer costs almost nothing (tagging customer feedback, categorizing a log entry), accuracy can be 60–70%. If a wrong answer is expensive (approving a loan, diagnosing a medical condition), accuracy needs to be 85%+.

Layer in latency constraints. Do you have 1 second, 5 seconds, or 30 seconds? Real-time systems eliminate chain-of-thought. Batch systems can afford it.

Add token budget. How many tokens can you burn per request? High-volume, low-stakes = minimize tokens. Low-volume, high-stakes = token budget is secondary.

Consider explainability requirements. Do stakeholders need to understand why the model made a decision? If yes, chain-of-thought or few-shot are mandatory. Zero-shot produces opaque outputs.

A decision tree:

• Accuracy <75%, Latency <2s, High volume? → Zero-shot. Simple task, straightforward input/output. Example: “Is this feedback positive or negative?”
• Accuracy 75–85%, Latency <5s, Medium volume? → Few-shot. Specific output format or domain terminology matters. Example: “Classify this invoice by vendor category.”
• Accuracy 80–90%, Latency <10s, Low-to-medium volume? → Chain-of-thought. Multi-step reasoning. Example: “Assess customer churn risk based on this profile.”
• Accuracy 85%+, Latency <15s, Low volume, high stakes? → Few-shot + Chain-of-thought. Critical decisions where explainability matters. Example: “Approve or deny this loan application.”

Common Pitfalls and How to Avoid Them

Pitfall 1: Assuming more examples always improve few-shot.

They don’t. After 5 examples, accuracy gains plateau or reverse. Why? The prompt becomes so long that the model loses focus. It’s attending to examples instead of the actual task. Test with 2, 3, 4, and 5 examples. Measure accuracy for each. Pick the sweet spot — usually 3.

Pitfall 2: Poor example selection.

If you pick examples that are easier than real data, or unrepresentative of the actual distribution, the model learns the wrong pattern. Always test your examples on a holdout set that matches production data. I’ve seen teams use “nice” examples for prompts and watch accuracy collapse when real messy data hits.

Pitfall 3: Chain-of-thought that doesn’t actually reason.

If you ask “Explain your reasoning” but don’t structure the reasoning (no step 1, 2, 3), the model produces text that sounds like reasoning but isn’t. It can be hallucination dressed up as explanation. Always provide a reasoning template: “1. [Factor]. 2. [Factor]. 3. [Factor]. Then: Decision.”

Pitfall 4: Confusing few-shot with RAG.

Few-shot puts examples in the prompt. RAG (retrieval-augmented generation) retrieves relevant documents and uses them as context. They’re not the same. Few-shot is good for teaching the model a specific output format or decision logic. RAG is good for grounding the model in fact-based information. Often you need both.

Pitfall 5: Not measuring latency impact.

Chain-of-thought adds real latency. I measured it with Claude Sonnet 4: zero-shot takes 0.8 seconds, few-shot (3 examples) takes 2.1 seconds, chain-of-thought takes 3.4 seconds, few-shot + CoT takes 5.2 seconds. If your SLA is 2 seconds, CoT is a non-starter. Measure before committing.

Building Your Prompting Stack: A Practical Approach

In AlgoVesta, we use all three techniques. Not in isolation — layered based on the task.

Here’s how:

Layer 1 — Triage with zero-shot. When a trading signal comes in, we first classify it (bullish, bearish, neutral) with a zero-shot prompt. This takes 300 tokens, 1 second. 80% of signals are clear-cut. They’re routed immediately.

Layer 2 — Deep analysis with few-shot + CoT. The remaining 20% (ambiguous or high-stakes signals) get few-shot prompting with 3 examples of similar ambiguous cases, plus chain-of-thought reasoning. This takes 1,800 tokens, 4 seconds. But for a signal that might move millions in allocation, 4 seconds and 1,800 tokens is trivial.

Layer 3 — Human review for edge cases. Signals where the model’s confidence is below 70% go to a human analyst. The model provides its step-by-step reasoning, so the human doesn’t start from scratch.

This tiered approach optimizes for speed, cost, and accuracy simultaneously. Most volume is handled cheaply and quickly. High-stakes decisions get the full treatment.

You can apply the same logic to your system:

1. Identify the volume distribution of your task. What percentage of requests are straightforward? What percentage are ambiguous or complex?
2. Route straightforward requests through zero-shot.
3. Route ambiguous or complex requests through few-shot + CoT.
4. For the hardest cases (high stakes, truly novel), add human review with AI assistance.
5. Measure accuracy at each layer. Adjust example quality and reasoning prompts based on failure modes.

This scales.

One Action to Take This Week

Pick one production prompt in your system. The one that handles the most volume or the highest stakes.

Measure its current accuracy. Then run three parallel experiments:

1. Keep it as is (baseline). Document the accuracy rate.
2. Add 3 few-shot examples. Choose examples that are representative of your actual data distribution — not cherry-picked. Measure accuracy.
3. Add chain-of-thought reasoning. Use the same baseline prompt, but ask the model to explain step-by-step before answering. Measure accuracy and latency.

Compare the three. Which gives you the best accuracy-to-latency-to-token-cost ratio? That’s your answer for that specific task.

Most teams do this once and never iterate. The better move is to repeat this quarterly. Your data evolves. Your edge cases shift. Your prompts should too.