You’ve built a RAG system. Documents go in, embeddings happen, similarity search returns relevant chunks. It works. Then your dataset grows from 100 documents to 100,000. Search latency balloons from 200ms to 8 seconds. The Chroma instance you spun up on your laptop can’t scale. Now you’re asking: do I actually need a dedicated vector database, or is this a sales pitch?
What a Vector Database Actually Does
Let’s separate hype from function. A vector database is a specialized system optimized for one job: storing high-dimensional vectors (embeddings) and returning the k most similar vectors to a query in sub-second time. That’s it. No transactions. No complex joins. No ACID guarantees. Just fast nearest-neighbor search at scale.
Standard relational databases can store vectors. PostgreSQL has pgvector extension. It works. But “works” and “works well at scale” are different things. A PostgreSQL query scanning 10 million embedding vectors with brute-force cosine similarity will run in 2–5 seconds. A vector database on the same dataset returns results in 50–200ms. The difference is algorithmic — vector databases use index structures (HNSW, IVF, DiskANN) designed specifically for approximate nearest-neighbor search, not exact matching.
Translation: if you have fewer than 50,000 embeddings and sub-500ms latency is acceptable, you don’t need this. If you have millions of vectors, production traffic, and users waiting for search results, you do.
Pinecone vs. Weaviate vs. ChromaDB: The Real Tradeoffs
Pinecone: Serverless, fully managed, cloud-only. You send vectors via API. Pinecone handles scaling, replication, backups. Zero infrastructure. Cost: $0.25–$1.50 per 1 million vectors per month depending on the $0.04 per 1K queries on top. Good for: teams that don’t want to manage databases. Limitations: vendor lock-in, no local development version, slower than self-hosted alternatives due to network latency.
Weaviate: Open-source, self-hosted or managed cloud version. You run the database (on your servers or Weaviate Cloud). Full control over deployment, data residency, scaling. Built-in support for hybrid search (vector + keyword filtering). Better for: teams with specific compliance requirements, preference for open-source, or existing Kubernetes infrastructure. Trade: you manage upgrades, backups, and scaling yourself. Latency is lower than Pinecone because queries don’t cross the internet.
ChromaDB: Lightweight, open-source, designed for prototyping. Runs in-process (no server) or as a standalone service. Stores data locally or in cloud storage. Best for: experimentation, small datasets (under 100k vectors), development environments. Not production-ready at scale — latency degrades fast beyond 500k vectors, and there’s limited distributed query support.
Real numbers: in a benchmark test with 1 million OpenAI embeddings (1536 dimensions), Pinecone returned results in ~120ms, Weaviate self-hosted in ~80ms, ChromaDB in ~600ms. Network latency to Pinecone’s API adds 50–100ms depending on geography. That matters when you’re making multiple queries per user request.
When You Absolutely Need a Vector Database
Three scenarios:
- Scale + latency pressure: More than 100k embeddings + user-facing search that needs to complete in under 500ms. PostgreSQL + pgvector will work, but not fast.
- Hybrid search: You need to filter vectors by metadata before similarity search (“find documents similar to X, but only from 2024”). Vector databases have native filtering. Doing this in PostgreSQL requires a separate WHERE clause that defeats index optimization.
- Real-time updates: You add/remove documents constantly. Pinecone and Weaviate support upserts without full re-indexing. Rebuilding ChromaDB or PostgreSQL indices gets expensive at scale.
A Practical Setup: When to Use What
Start with ChromaDB if your dataset is under 10k documents and latency isn’t a constraint. Deploy it in-process, store vectors in JSON, move forward. You’ll spend 0 on infrastructure and know immediately if vector search solves your problem.
Move to Pinecone when you hit one of these walls: you need sub-200ms latency, your dataset grows beyond 100k vectors, or you don’t want to manage infrastructure. The $0.04 per 1K queries charge adds up at scale, but you’re paying for speed and managed reliability. No index tuning, no capacity planning.
Choose Weaviate if you’re already running Kubernetes, need to host vectors in your own cloud account, or require custom hybrid search logic. You’re trading convenience for control. The setup takes a week. Scaling takes maintenance. But you own the data and the latency is better.
The Code Reality: Embedding Storage
Here’s what a basic embedding + search workflow looks like in ChromaDB (development) vs. Pinecone (production scale):
# ChromaDB: Simple, in-process
import chromadb
from chromadb.utils import embedding_functions
client = chromadb.Client()
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
api_key="your-key",
model_name="text-embedding-3-small"
)
collection = client.get_or_create_collection(
name="docs",
embedding_function=openai_ef
)
# Add documents
collection.add(
ids=["doc1", "doc2"],
documents=["Content A", "Content B"]
)
# Search
results = collection.query(
query_texts=["Find similar content"],
n_results=3
)
print(results['documents'][0])# Pinecone: Scalable, API-based
import pinecone
from openai import OpenAI
pinecone.init(api_key="your-key", environment="us-west-2-aws")
index = pinecone.Index("production-index")
client = OpenAI(api_key="your-key")
# Embed and store
docs = ["Content A", "Content B"]
embeddings = [
client.embeddings.create(
input=doc,
model="text-embedding-3-small"
).data[0].embedding
for doc in docs
]
# Upsert (update or insert)
vectors = [
("doc1", embeddings[0], {"text": docs[0]}),
("doc2", embeddings[1], {"text": docs[1]})
]
index.upsert(vectors=vectors)
# Search
query_embedding = client.embeddings.create(
input="Find similar content",
model="text-embedding-3-small"
).data[0].embedding
results = index.query(
vector=query_embedding,
top_k=3,
include_metadata=True
)
for match in results['matches']:
print(match['metadata']['text'])The Chroma version is 8 lines. Pinecone requires API keys, embeddings computed separately, and structured metadata. But Pinecone scales to millions of vectors without degradation. Chroma slows down visibly beyond 500k.
Do This Today: Test Your Scale Ceiling
Before choosing a database, run ChromaDB with your actual document count. Measure query latency at 10k vectors, 100k, and 1 million (if feasible). Set a latency threshold — maybe 300ms is acceptable for your users, maybe it isn’t. If ChromaDB hits that threshold before your data reaches 500k vectors, you have an answer. If it scales fine to your expected dataset size, stay local. The money you save on infrastructure beats vendor features you don’t need.
