Context Window Management: Fitting Long Documents Into LLMs

Your 50-page legal document gets uploaded to Claude, and the model returns analysis based on page 3. The rest exists in a token void. This isn’t a hallucination problem — it’s a context window problem, and it breaks more production systems than bad prompts do.

Context window is real estate. Every token you use — prompt, documents, conversation history, system instructions — occupies space that could go toward your actual task. Exceed the limit, and the model either stops responding or, worse, starts ignoring the oldest information to make room for new input. Understanding how to manage this space is the difference between a working system and one that mysteriously fails on Thursday when your document got longer.

Understanding Your Context Budget

Start by knowing exactly what you’re working with.

Claude 3.5 Sonnet gives you 200,000 tokens. GPT-4o gives you 128,000 tokens (or 4,096 if you’re on the older base model). Mistral Large offers 32,000 tokens. These numbers sound large until you start measuring real content.

A typical page of business text is roughly 250–350 tokens. A 50-page document costs you 12,500–17,500 tokens before you’ve written a single instruction. A long conversation with 20 exchanges burns 4,000–8,000 tokens. That leaves you with actual working space measured in what’s left over — your “processing budget.”

Calculate your exact token cost before deployment. Use the model provider’s tokenizer:

# Python example using Anthropic's tokenizer
import anthropic

client = anthropic.Anthropic()
with open('document.txt', 'r') as f:
    doc_text = f.read()

response = client.messages.count_tokens(
    model="claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": doc_text}]
)

print(f"Document tokens: {response.input_tokens}")
print(f"Remaining budget: {200000 - response.input_tokens}")

This isn’t optional. Guessing costs you failed queries and wasted API spend.

The Three Compression Techniques That Actually Work

You have three levers: extract what matters, summarize strategically, or break the work into chunks. Most people try to skip straight to summarization and fail because they’re summarizing the wrong parts.

1. Extraction Before Summarization

Don’t summarize everything. Extract the sections relevant to your actual question first.

If you’re analyzing a contract for payment terms, you don’t need summaries of the company history or product specifications. Pull the relevant clauses, pass those to the LLM, and keep 80% of your context window free.

Bad approach:

# Summarize the entire 50-page document
Prompt: "Summarize this legal document in 500 words."
[50-page document]
"What are the payment terms?"

Good approach:

# Extract relevant sections, then answer
Prompt: "Extract all sections mentioning 'payment', 'invoice', 'due date', 
or 'terms'. Keep the exact wording."  
[50-page document]

# Then in a second call or same call with extracted text:
Prompt: "Based on these sections, what are the payment terms and when 
are invoices due?"
[Extracted sections only - ~2000 tokens instead of 15000]

Use keyword extraction, regex patterns, or a lightweight search pass (Claude Mini on shorter docs) to identify relevant chunks before passing to your main analysis model.

2. Hierarchical Summarization

If extraction isn’t viable — say, you need the whole context — summarize in layers. Don’t summarize 50 pages to 500 words in one pass. Summarize chunks to summaries, then summarize those.

Break the document into 5–10 page chunks. Summarize each to 150–200 tokens. Then summarize those summaries. You compress 15,000 tokens down to 500–800 while preserving structure and relationships.

This preserves nuance better than one aggressive summary because the model never discards detail arbitrarily — it condenses proportionally at each layer.

3. Sliding Window Processing for Conversations

Long conversations fill context fast. Don’t keep the entire history. Keep recent exchanges (last 5–10) plus summaries of older ones.

# Conversation context management
conversation = [
    {"role": "user", "content": "Tell me about AWS pricing"},
    {"role": "assistant", "content": "AWS pricing is..."}
    # ... 15 more exchanges ...
]

# Before API call, trim and summarize
if conversation_token_count > 100000:
    # Keep last 5 exchanges
    recent = conversation[-10:]  # 5 user/assistant pairs
    
    # Summarize everything before that
    older_summary = summarize_conversation(conversation[:-10])
    
    # Reconstruct with explicit history marker
    trimmed = [
        {"role": "system", 
         "content": "Earlier in this conversation: " + older_summary}
    ] + recent

Temperature and Sampling: Hidden Context Problems

You can also manage context efficiency through model behavior, not just input size.

Lower temperature (0.3–0.5) makes the model focus on what it already knows — useful for fact extraction from provided context. Higher temperature (0.7–1.0) increases randomness, which burns tokens on less predictable outputs.

For context-heavy tasks (“answer only from this document”), run at 0.3–0.4. The model’s attention stays on your input, not on exploring alternative phrasings.

Max tokens parameter matters too. Set it deliberately. If you’re extracting payment terms, you don’t need 4,096 token outputs. Set max_tokens=500. This reserves budget for your actual task instead of letting the model ramble into empty space.

The Real Setup: A Production Pattern

Here’s what actually works at scale:

1. Extraction layer: Use regex, keyword search, or a fast model (Claude 3.5 Haiku) to pull relevant sections from large documents. Cost: minimal.
2. Token budget check: Count tokens of extracted sections. If under 30,000 tokens with Claude Sonnet, proceed. If not, summarize.
3. Summarize if needed: Hierarchical summaries of chunks, not aggressive single-pass compression.
4. Query with context: Pass extracted or summarized material plus explicit instructions: “Answer only based on these sections. Do not use external knowledge.”
5. Conversation trim: After 10 user messages, snapshot the conversation, summarize it, and start a new window.

The bottleneck is rarely the model. It’s the prep work before the API call.

Start With Token Counting, Not Summarization

Today: measure your actual documents. Run them through the tokenizer for the model you’re using. Calculate how many tokens remain after your document uploads. That remaining number is your actual working budget.

If it’s under 30,000 for Claude Sonnet or under 50,000 for production work, you need to extract, not summarize. If it’s negative, you need a different approach entirely — split documents, use vector search to retrieve only relevant sections, or use a model with a larger window.

Most context problems aren’t solved by better prompts. They’re solved by not asking the model to process more than it needs to.