Building AI Agents: Tool Calling, Memory, and Loop Patterns

Most developers treat AI agents like chatbots with extra steps. Ask a question, get an answer, move on. The moment you need an agent to actually do something — fetch data, update a database, make a decision across multiple steps — that model breaks. This is where architecture matters, and where most implementations fail within the first two weeks of production.

An AI agent is a loop, not a model. The model is the decision engine. The loop is the operating system that decides what happens next.

The Core Agent Loop

Every functional agent follows this pattern:

1. User provides input/context
2. LLM decides what to do (including: do nothing)
3. If the LLM chose an action:
   - Execute the tool
   - Capture the result
   - Return result to LLM
4. Repeat until the LLM says "I'm done"
5. Return final response to user

The loop is the contract. The model is the decision-maker inside it. Confuse these, and you’ll waste weeks debugging prompts when the real problem is your loop logic.

I learned this at AlgoVesta the hard way. We built an agent to analyze market data and execute trades. The prompt was locked in at 40% accuracy — until we realized the loop was calling the same tool twice, feeding it stale results from the first call, then wondering why the agent made bad decisions. The model was fine. The plumbing was broken.

Tool Calling: The Contract Between LLM and Code

Tool calling is how the LLM tells your code what to execute. It’s not a prompt technique. It’s an API contract.

Most models support it natively now — Claude (via tool_use block), GPT-4o (via function_calling), Mistral (via tool_call). The names differ. The concept is identical: the model returns structured data that says “run this tool with these parameters.”

Here’s what a basic tool definition looks like:

{
  "name": "fetch_user_data",
  "description": "Retrieves user account info including balance and transaction history",
  "input_schema": {
    "type": "object",
    "properties": {
      "user_id": {
        "type": "string",
        "description": "The unique user identifier"
      },
      "include_history": {
        "type": "boolean",
        "description": "Include transaction history (default: false)"
      }
    },
    "required": ["user_id"]
  }
}

The description matters. A vague description like “get data” results in the model using the tool wrong. A specific description like “Retrieves user account info including balance and transaction history” gives the model context to decide whether it needs this tool at all.

Here’s a real scenario: we had an agent that was supposed to check user eligibility before making decisions. It kept calling the wrong tool because the description was generic. Changed “Validate customer eligibility based on account age, balance, and transaction patterns” and the error rate dropped from 18% to 3%.

The tool definition is half prompt engineering. Write it clearly.

Memory: Conversation or State

This is where most hobby projects diverge from production systems.

Conversation memory (the chat history you feed back to the model) works until it doesn’t. Token limits exist. Claude Sonnet 4 has 200k tokens, but feeding a 6-month conversation history into every API call wastes tokens and slows inference. After AlgoVesta hit ~3,000 agent interactions per month, we realized we were burning budget on context the model didn’t need.

Production agents need two memory layers:

Short-term memory: The current conversation or task. Keep it tight — only the last 5–10 messages, or the last 5 minutes of interaction, whichever is smaller.

Long-term memory: Facts the agent needs to remember but doesn’t need in every prompt. Store this separately — a database, a vector store, or a structured knowledge base — and retrieve it only when relevant.

Here’s the pattern:

1. User sends message
2. Query long-term memory for relevant facts
3. Add those facts to the system prompt
4. Add recent conversation history (last N messages)
5. Send to LLM
6. If the agent learned something important, store it
7. Proceed with tool calling

For a trading agent, we store prior decisions and their outcomes. When the agent is deciding whether to execute a trade, we retrieve the last 5 similar trades and their results — not the entire conversation history, just the signal.

This is a 10-line change from “naive memory” to “scalable memory.” Most developers never make it.

Failure Handling and Retry Logic

A tool call fails. The database was slow. The API returned a timeout. What does the agent do?

If your loop just crashes, you’ve built a toy. Production agents need fallback logic.

Minimal viable pattern:

for attempt in range(max_retries):
    try:
        result = execute_tool(tool_name, params)
        if result.success:
            return result
    except ToolExecutionError as e:
        if attempt == max_retries - 1:
            # Final attempt failed. Tell the LLM.
            agent_message = f"Tool '{tool_name}' failed: {e}. Choose another approach."
            # Feed this back to the agent, let it decide next step
        else:
            time.sleep(2 ** attempt)  # exponential backoff
            continue

The critical line: tell the LLM that the tool failed, and let it decide what to do. It might retry, choose a different tool, or report the error to the user. You don’t decide — the agent does.

Putting It Together: A Minimal Agent Implementation

Start here. This is the shape of a real agent:

def run_agent(user_input, system_prompt, tools):
    messages = [{"role": "user", "content": user_input}]
    max_iterations = 10
    iteration = 0
    
    while iteration < max_iterations:
        iteration += 1
        
        # Call LLM with current messages and available tools
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=1024,
            system=system_prompt,
            tools=tools,
            messages=messages
        )
        
        # Check if LLM is done
        if response.stop_reason == "end_turn":
            return extract_text_response(response)
        
        # Process tool calls
        if response.stop_reason == "tool_use":
            tool_results = []
            for block in response.content:
                if block.type == "tool_use":
                    result = execute_tool(block.name, block.input)
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": json.dumps(result)
                    })
            
            # Add LLM response and tool results to messages
            messages.append({"role": "assistant", "content": response.content})
            messages.append({"role": "user", "content": tool_results})
    
    return "Agent exceeded max iterations"

This runs on Claude Sonnet 4. It's not production-hardened — add timeouts, logging, error handling — but it's the actual shape. The loop is explicit. The tool calling is visible. Memory is just the messages list (short-term only, for now).

Start Simple, Then Optimize

Build this loop first. Get tool calling working. Make sure your tools are discoverable and your descriptions are precise. Once the agent can execute a task reliably, add memory management and failure handling.

Most developers try to build the "perfect" agent from day one. You can't. You have to know what breaks first.

Take the loop above. Add two or three real tools you actually need. Run it for a day. Where did it fail? That's where you add complexity next.