Your conversation with Claude hits 80,000 tokens. The model starts repeating itself. GPT-4o suddenly forgets context from five messages back. Mistral 7B on your local machine begins hallucinating details that were mentioned earlier.
These aren’t random failures. They’re symptomatic of context window mismanagement — the gap between what a model can theoretically hold and what it actually uses effectively.
Understanding Context Window Limits (And What They Actually Mean)
A context window is the amount of text — measured in tokens — that a model can consider when generating a response. Claude 3.5 Sonnet supports 200,000 tokens. GPT-4o supports 128,000. Llama 3 70B supports 8,000 in its base version.
But having a 200,000-token window doesn’t mean you should use all 200,000 tokens for your conversation.
Models perform worse on tasks when the window fills up — especially on retrieval tasks where they need to find specific information buried in earlier context. Anthropic’s internal testing shows that Claude’s accuracy on “needle in a haystack” retrieval tasks (finding a specific fact in a long document) drops roughly 5-7% for every 25% of the window you fill. At 80% capacity, you’re looking at degraded performance on information recall, even though tokens still fit.
The practical window — where the model performs reliably — is usually 60-70% of the theoretical maximum. Beyond that, accuracy decays noticeably.
The Three Strategies That Actually Work
1. Summarization Before Compression
Don’t just truncate old messages. Summarize them.
When a conversation grows beyond 40,000 tokens (for Claude Sonnet) or 30,000 tokens (for GPT-4o), stop and create a summary of everything discussed so far. This serves two purposes: it preserves semantic meaning without the token bloat, and it forces the model to consolidate its own understanding.
# Bad approach: just keep adding messages
User: [Message 1]
Assistant: [Response 1]
User: [Message 2]
Assistant: [Response 2]
... repeat 50 times ...
User: [Message 51 - runs out of context window]
# Better approach: summarize at checkpoints
User: [Message 1-10]
Assistant: [Response]
User: Please summarize our conversation so far
Assistant: [Summary of discussion, key decisions, context]
# Now append new messages to the summary, not the full history
Context: [Summary from above]
User: [Message 11]
Assistant: [Response using both summary and new message]The summary becomes the new “context base” for subsequent messages. You’ve compressed 10 messages into 200-400 tokens while retaining 95%+ of the semantic value.
2. Sliding Window with Explicit Context Injection
For applications where you can’t pause and summarize — like a chatbot that needs to respond in real-time — use a sliding window approach. Keep only the last N messages in active context, plus a fixed system instruction that defines the interaction style.
# System instruction (always included, counts as context)
You are a technical advisor. When the user asks about deployment,
remember: we use AWS. When discussing testing, reference the
existing test suite in the codebase.
# Sliding window: keep last 5 messages only
[Previous messages deleted]
User: [Message N-4]
Assistant: [Response]
User: [Message N-3]
Assistant: [Response]
User: [Message N-2]
Assistant: [Response]
User: [Message N-1]
Assistant: [Response]
User: [Message N] <- incoming
# Token usage: system instruction + last 5 messages
# Result: ~4,000-6,000 tokens depending on message lengthThe trade-off is clear: you lose historical context beyond the last 5 messages, but you maintain consistent performance. For use cases where users don't reference things from 20 messages ago — customer support, code review, iterative design — this works well.
3. Retrieval-Augmented Context (RAG Pattern)
If you need access to old context without keeping it all in the conversation, embed and index previous messages or documents, then retrieve only the relevant ones.
Instead of passing the full 40,000-token conversation to the model, you:
- Convert each message or section into an embedding
- Store embeddings in a vector database (Pinecone, Weaviate, even SQLite with vector extension)
- When the user sends a new message, retrieve the top 3-5 most similar previous messages
- Inject those into context, along with the current message
This keeps your active context window to 5,000-8,000 tokens while giving access to an effectively unlimited conversation history. The model only sees what's relevant to the current query.
# Pseudocode for RAG-based context management
import anthropic
from embedding_service import embed_and_store, retrieve_similar
def chat_with_rag_context(user_message, conversation_id):
# Retrieve similar past messages
similar_messages = retrieve_similar(
query=user_message,
conversation_id=conversation_id,
limit=4
)
# Build context window
context = "Previous relevant messages:\n"
for msg in similar_messages:
context += f"- {msg['content']}\n"
# Add current message
context += f"\nCurrent question: {user_message}"
# Send to Claude with bounded context
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
messages=[
{"role": "user", "content": context}
]
)
return response.content[0].textWhen Each Strategy Fails
Summarization breaks down if your domain requires precision on numerical data — a summary might say "we discussed pricing" but lose the exact figures. Use this for narrative context, not quantitative detail.
Sliding windows fail for multi-turn reasoning tasks where the model needs to reference decisions from 15+ messages back. If your use case involves step-by-step problem-solving with frequent back-references, you'll need either higher capacity or active summarization.
RAG fails when relevance is hard to compute — if you're asking "given everything we've discussed, what's the best next step?" a keyword/embedding search might miss the subtle context that shaped earlier decisions.
The Immediate Action
Pick one: if you're building a chatbot, implement the sliding window approach today. Set a hard limit of 6 recent messages + a 200-word system instruction. If you're working with document processing or long-form analysis, test summarization at the 40,000-token mark for your chosen model. Track whether accuracy on retrieval tasks improves when you compare full-context runs against summarized runs.
Context window size matters less than context window discipline.
