Running Llama 3, Mistral, and Phi Locally: Hardware Setup and First Inference

You can run a capable language model on your laptop right now. Not a toy model — something that handles real tasks. Llama 3 8B, Mistral 7B, and Phi 3.5 all run on consumer hardware. The barrier isn’t capability anymore. It’s knowing which tool, which quantization, and which 20 minutes of setup actually work.

Why Local LLMs Matter Now

For three years, local inference meant either using a 7B parameter model that performed like GPT-3, or burning $200 a month on API calls. That changed in 2024. Llama 3 8B (released July 2024) performs at GPT-3.5 level on most tasks. Mistral 7B beats GPT-3.5 on reasoning. Phi 3.5 runs on 8GB RAM and handles summarization, classification, and code review without obvious degradation.

The practical upside: zero API costs for development, inference latency under 500ms on modest hardware, and your data never leaves your machine. The catch: you need the right quantization, the right tool, and realistic expectations about speed versus quality tradeoffs.

Hardware Reality Check

Before downloading anything, you need to know what you’re working with.

• RAM: A 7B model at 4-bit quantization needs roughly 6–8GB. At 8-bit: 14–16GB. Unquantized (float32): 30GB minimum. You’re probably running 4-bit.
• GPU: Optional but transformative. An RTX 4060 (8GB VRAM) runs Llama 3 8B at 4-bit with 30+ tokens/second. Without GPU: CPU inference on an M1 MacBook Pro does 5–8 tokens/second — acceptable for batch processing, rough for interactive use.
• Disk: A 7B model quantized to 4-bit is 3–5GB. Keep 20GB free for model downloads and workspace.

If you’re on a MacBook Air M1/M2, stop here — Ollama (below) handles all GPU acceleration automatically. If you’re on Windows or Linux with an RTX card, GPU acceleration cuts inference time by 4–6x.

Ollama: The Zero-Friction Path

If you want to run a model in the next five minutes without wrestling with Python environments, use Ollama.

Download from ollama.ai, run the installer. Then:

ollama run llama2

That’s it. Ollama auto-downloads, quantizes, and serves the model on localhost:11434. You now have a local API compatible with OpenAI’s chat interface.

To run Mistral instead:

ollama run mistral

Ollama pulls the recommended quantization (usually Q4_K_M — 4-bit, medium variant) automatically. No config files. No CUDA wrangling on Linux. On Mac, it detects GPU automatically.

The limitation: Ollama abstracts away quantization choices. If you need Q2_K (ultra-low VRAM) or Q8_0 (maximum quality), you’ll need the next approach.

LM Studio: Control and Simplicity

For 80% of practitioners, Ollama suffices. For the remaining 20% — people running on 4GB RAM, or chasing specific quality/speed tradeoffs — LM Studio is the next step.

LM Studio gives you a GUI, quantization picker, and the same OpenAI-compatible API as Ollama.

Install from lmstudio.ai. Open the app. Search for “Mistral 7B”. You’ll see 15 versions:

• Q2_K: 3.5GB, ~2 tokens/sec on CPU
• Q4_K_M: 5GB, ~5 tokens/sec on CPU
• Q6_K: 8GB, ~3 tokens/sec on CPU (higher quality than Q4)
• Q8_0: 14GB, near-original quality, slower

For most work, Q4_K_M is the default answer. It’s the sweet spot between quality and resource use that Ollama also defaults to.

Download the model, click “Load”, then use it via API:

import requests
import json

response = requests.post(
    "http://localhost:1234/v1/chat/completions",
    json={
        "model": "local-model",
        "messages": [{"role": "user", "content": "Summarize this in one sentence: [your text]"}],
        "temperature": 0.7
    }
)
print(response.json()['choices'][0]['message']['content'])

The API is identical to OpenAI’s. That matters — you can test locally, then swap the endpoint to GPT-4o without rewriting code.

When to Pick Which Model

Llama 3 8B, Mistral 7B, and Phi 3.5 solve different problems at different resource levels.

• Phi 3.5 Mini (3.8B params, 2GB quantized): Runs on any hardware. Best for classification, extraction, summarization. Loses coherence on open-ended generation past 1000 tokens.
• Mistral 7B (7B params, 5GB quantized): Strongest reasoning for its size. Better than Llama 3 on code and structured output. Slightly weaker on creative writing.
• Llama 3 8B (8B params, 6GB quantized): Most balanced. Good at everything. Slower than Mistral on CPU (larger parameter count), but more reliable on long documents.

If you have 8GB RAM: start with Phi 3.5. If you have 16GB or a GPU: Mistral 7B. If you need maximum flexibility: Llama 3 8B.

Test It Today: One Working Example

Install Ollama. Run this:

ollama run mistral

Wait 3–4 minutes for the download. Then paste this into the prompt:

Extract the entities from this text as JSON. Return only valid JSON, no explanation.

Text: Apple released the iPhone on June 29, 2007. Steve Jobs presented it in San Francisco.

Return format: {"companies": [], "products": [], "dates": [], "people": []}

You’ll get valid JSON back in under 5 seconds on most hardware. That’s extraction — not hallucination, not generation drift. A real task that used to require an API call. Now it’s local, free, and offline.

That prompt works because it’s constrained (JSON format enforces structure) and Mistral 7B is strong on instruction-following. If you try the same prompt on Phi 3.5, you might get malformed JSON occasionally — a quantitative difference worth knowing before you build on it.

Next: Integrate Into Your Workflow

Set Ollama or LM Studio to run on startup. Add this to your Python environment for any extraction or classification pipeline:

from openai import OpenAI

client = OpenAI(api_key="not-used", base_url="http://localhost:11434/v1")

def classify_text(text, categories):
    response = client.chat.completions.create(
        model="mistral",
        messages=[{
            "role": "user",
            "content": f"Classify this text as one of: {', '.join(categories)}\n\nText: {text}"
        }]
    )
    return response.choices[0].message.content

That’s your local inference layer. It’s API-compatible with OpenAI, zero-cost, and ready for production use on repetitive tasks where speed-of-light latency doesn’t matter — batch processing, background jobs, classification pipelines.